Communication management apparatus, image communication system, communication management method, and recording medium

ABSTRACT

A communication management apparatus includes circuitry to receive from a communication terminal predetermined-position information indicating a predetermined position in an area where a plurality of image capturing devices are provided, and number-of-division information indicating the number of divisions for dividing the area from the predetermined position, and transmit to the communication terminal first predetermined-area information indicating a predetermined area including the predetermined position in an image captured by a first image capturing device and second predetermined-area information indicating a predetermined area including the predetermined position in an image captured by a second image capturing device, the first image capturing device being arranged at a shortest distance from the predetermined position, and the second image capturing device being arranged such that an angle between the second and the first image capturing devices is closest to an angle obtained by dividing the area in accordance with the number of divisions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35U.S.C. § 119(a) to Japanese Patent Application No. 2020-166189, filed onSep. 30, 2020, in the Japan Patent Office, the entire disclosure ofwhich is hereby incorporated by reference herein.

BACKGROUND Technical Field

The present disclosure relates to a communication management apparatus,an image communication system, a communication management method, and arecording medium.

Description of the Related Art

An image capturing device capable of capturing an omnidirectional sceneusing a plurality of wide-angle lenses or fish-eye lenses is known. Alsoknown is a system capable of distributing image data of an imagecaptured using such an image capturing device in real time such that adistribution site where the image capturing device is disposed isviewable at a different site in real time.

For example, there is a system in which images captured by a pluralityof cameras at a remote location are displayed on a terminal such that auser can grasp the situation at the remote location.

In the existing method, however, a user who desires to view a scene at apredetermined position in an area where a plurality of image capturingdevices are disposed, individually operates a plurality of imagecapturing devices in an area including the predetermined position.

SUMMARY

Example embodiments include a communication management apparatus forcommunicating with a communication terminal that displays imagescaptured by a plurality of image capturing devices. The communicationmanagement apparatus includes circuitry that receives, from thecommunication terminal, predetermined-position information indicating apredetermined position in an area where the plurality of image capturingdevices are provided and number-of-division information indicating thenumber of divisions for dividing the area from the predeterminedposition. The circuitry further transmits, to the communicationterminal, first predetermined-area information indicating apredetermined area including the predetermined position in an imagecaptured by a first image capturing device among the plurality of imagecapturing devices, and second predetermined-area information indicatinga predetermined area including the predetermined position in an imagecaptured by a second image capturing device among the plurality of imagecapturing devices. The first image capturing device is arranged at ashortest distance from the predetermined position corresponding to thepredetermined-position information. the second image capturing device isarranged such that an angle between the second image capturing deviceand the first image capturing device is closest to an angle obtained bydividing the area in accordance with the number of divisions indicatedby the number-of-division information.

Example embodiments include an image communication system including theabove-described communication management apparatus, and a communicationterminal configured to display images captured by a plurality of imagecapturing devices.

Example embodiments include a communication management method executedby a communication management apparatus, the communication managementapparatus is communicable with a communication terminal that displaysimages captured by a plurality of image capturing devices. Thecommunication management method includes: receiving, from thecommunication terminal, predetermined-position information indicating apredetermined position in an area where the plurality of image capturingdevices are provided and number-of-division information indicating thenumber of divisions for dividing the area from the predeterminedposition; and transmitting, to the communication terminal, firstpredetermined-area information indicating a predetermined area includingthe predetermined position in an image captured by a first imagecapturing device among the plurality of image capturing devices, andsecond predetermined-area information indicating a predetermined areaincluding the predetermined position in an image captured by a secondimage capturing device among the plurality of image capturing devices.The first image capturing device is arranged at a shortest distance fromthe predetermined position corresponding to the predetermined-positioninformation. The second image capturing device is arranged such that anangle between the second image capturing device and the first imagecapturing device is closest to an angle obtained by dividing the area inaccordance with the number of divisions indicated by thenumber-of-division information.

Example embodiments include a non-transitory recording medium which,when executed by one or more processors, cause a communicationmanagement method.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages and features thereof can be readily obtained and understoodfrom the following detailed description with reference to theaccompanying drawings, wherein:

FIGS. 1A, 1B, and 1C are a side view, a front view, and a plan view ofan image capturing device according to an embodiment of the presentdisclosure, respectively;

FIG. 2 is an illustration for explaining how a user uses the imagecapturing device, according to an embodiment of the present disclosure;

FIGS. 3A, 3B, and 3C are views illustrating a hemispherical image (frontside) captured by the image capturing device, a hemispherical image(back side) captured by the image capturing device, and an image inequirectangular projection, respectively, according to an embodiment ofthe present disclosure;

FIG. 4A is a conceptual diagram illustrating how a surface of a sphereis covered with the equirectangular projection image according to anembodiment of the present disclosure;

FIG. 4B is a view illustrating a spherical image according to anembodiment of the present disclosure;

FIG. 5 is a view illustrating positions of a virtual camera and apredetermined area in a case in which the spherical image is of athree-dimensional solid sphere according to an embodiment of the presentdisclosure;

FIG. 6A is a perspective view of the virtual camera and thepredetermined area illustrated in FIG. 5 according to an embodiment ofthe present disclosure;

FIG. 6B is a view illustrating an image of the predetermined areadisplayed on a display according to an embodiment of the presentdisclosure;

FIG. 7 is a view illustrating a relationship between predetermined-areainformation and the image of the predetermined area according to anembodiment of the present disclosure;

FIG. 8 is a view illustrating a point in a three-dimensional Euclideanspace according to spherical coordinates, according to an embodiment ofthe present disclosure;

FIG. 9 is a diagram illustrating the general arrangement of an imagecommunication system according to an embodiment of the presentdisclosure;

FIG. 10 is a view illustrating an overview of a distribution site in theimage communication system according to an embodiment of the presentdisclosure;

FIG. 11 is a diagram illustrating an example hardware configuration ofthe image capturing device according to an embodiment of the presentdisclosure;

FIG. 12 is a diagram illustrating an example hardware configuration of adistribution terminal, a communication management apparatus, and acommunication terminal according to an embodiment of the presentdisclosure;

FIG. 13 is a diagram illustrating an example functional configuration ofthe image communication system according to an embodiment of the presentdisclosure;

FIG. 14 is a diagram illustrating an example functional configuration ofthe image communication system according to an embodiment of the presentdisclosure;

FIG. 15A is a conceptual diagram illustrating an example image capturingdevice management table according to an embodiment of the presentdisclosure;

FIG. 15B is a conceptual diagram illustrating an example image typemanagement table according to an embodiment of the present disclosure;

FIG. 16A is a conceptual diagram illustrating an example sessionmanagement table according to an embodiment of the present disclosure;

FIG. 16B is a conceptual diagram illustrating an example image typemanagement table according to an embodiment of the present disclosure;

FIG. 17A is a conceptual diagram illustrating an examplepredetermined-area management table according to an embodiment of thepresent disclosure;

FIG. 17B is a conceptual diagram illustrating an example arrangementinformation management table according to an embodiment of the presentdisclosure;

FIG. 18 is a conceptual diagram illustrating an exampledistribution-site management table according to an embodiment of thepresent disclosure;

FIG. 19A is a conceptual diagram illustrating an example weightingcoefficient management table according to an embodiment of the presentdisclosure;

FIG. 19B is a conceptual diagram illustrating an example specificweighting coefficient management table according to an embodiment of thepresent disclosure;

FIG. 20A is a conceptual diagram illustrating an example image typemanagement table according to an embodiment of the present disclosure;

FIG. 20B is a conceptual diagram illustrating an examplepredetermined-area management table according to an embodiment of thepresent disclosure;

FIG. 21 is a view schematically describing an example relationshipbetween a point of interest and image capturing devices when the numberof divisions is two, according to an embodiment of the presentdisclosure;

FIG. 22 is a view schematically describing an example selection processbased on an angle between image capturing devices when the number ofdivisions is two, according to an embodiment of the present disclosure;

FIG. 23 is a view schematically describing an example relationshipbetween a point of interest and image capturing devices when the numberof divisions is three, according to an embodiment of the presentdisclosure;

FIG. 24 is a view schematically describing an example selection processbased on an angle between image capturing devices when the number ofdivisions is three, according to an embodiment of the presentdisclosure;

FIG. 25 is a view schematically describing an example selection processbased on an angle between image capturing devices when the number ofdivisions is three, according to an embodiment of the presentdisclosure;

FIG. 26 is a view schematically describing an example relationshipbetween a point of interest and image capturing devices when the numberof divisions is four, according to an embodiment of the presentdisclosure;

FIG. 27 is a view schematically describing an example relationshipbetween a point of interest and image capturing devices in irregularpositions, according to an embodiment of the present disclosure;

FIG. 28 is a sequence diagram illustrating an example process forparticipating in a specific communication session in the imagecommunication system according to an embodiment of the presentdisclosure;

FIG. 29 is a view illustrating an example screen for selecting acommunication session according to an embodiment of the presentdisclosure;

FIG. 30 is a sequence diagram illustrating an example process formanaging image type information in the image communication systemaccording to an embodiment of the present disclosure;

FIG. 31 is a sequence diagram illustrating an example process forcommunicating captured image data and audio data in the imagecommunication system according to an embodiment of the presentdisclosure;

FIG. 32 is a sequence diagram illustrating an example process fordisplaying a point of interest in the image communication systemaccording to an embodiment of the present disclosure;

FIG. 33 is a flowchart illustrating an example process for selecting animage capturing device according to an embodiment of the presentdisclosure;

FIG. 34 is a view illustrating an example site display screen displayedon the communication terminal according to an embodiment of the presentdisclosure;

FIG. 35 is a flowchart illustrating an example process for selecting asupplementary image capturing device according to an embodiment of thepresent disclosure;

FIG. 36 is a view illustrating an example display screen on which apredetermined-area image is displayed, according to an embodiment of thepresent disclosure; and

FIG. 37 is a sequence diagram illustrating a modification of the processfor displaying a point of interest in the image communication system.

The accompanying drawings are intended to depict embodiments of thepresent invention and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted. Also, identical or similar referencenumerals designate identical or similar components throughout theseveral views.

DETAILED DESCRIPTION

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this specification is not intended to be limited to the specificterminology so selected and it is to be understood that each specificelement includes all technical equivalents that have a similar function,operate in a similar manner, and achieve a similar result.

Referring now to the drawings, embodiments of the present disclosure aredescribed below. As used herein, the singular forms “a,” “an,” and “the”are intended to include the plural forms as well, unless the contextclearly indicates otherwise.

Hereinafter, one or more embodiments of the present disclosure will bedescribed with reference to the drawings. In the drawings, any redundantdescriptions thereof will be omitted.

Method for Generating Spherical Image

A method for generating a spherical image according to one or moreembodiments will be described with reference to FIGS. 1A to 8.

First, the external appearance of an image capturing device 10 will bedescribed with reference to FIGS. 1A to 1C. The image capturing device10 is a digital camera for capturing images from which a 360-degreespherical image is generated. FIGS. 1A, 1B, and 1C are a side view, afront view, and a plan view of the image capturing device 10,respectively.

As illustrated in FIG. 1A, the image capturing device 10 has a size suchthat a person can hold the image capturing device 10 with one hand. Asillustrated in FIGS. 1A, 1B, and 1C, the image capturing device 10includes an imaging element 103 a and an imaging element 103 b in anupper portion thereof such that the imaging element 103 a is disposed onthe front side and the imaging element 103 b is disposed on the backside. The imaging elements (image sensors) 103 a and 103 b are usedtogether with an optical member (e.g., lenses 102 a and 102 b describedbelow) capable of capturing a hemispherical image (with an angle of viewof 180 degrees or more). As illustrated in FIG. 1B, the image capturingdevice 10 further includes an operation device 115 such as a shutterbutton on the back surface of the image capturing device 10.

Next, a situation in which the image capturing device 10 is used will bedescribed with reference to FIG. 2. FIG. 2 is an illustration forexplaining how a user uses the image capturing device 10. As illustratedin FIG. 2, the image capturing device 10 is used to, for example,capture an image of objects around the user when the user holds theimage capturing device 10 with one hand. The imaging elements 103 a and103 b illustrated in FIGS. 1A to 1C capture the objects surrounding theuser to obtain two hemispherical images.

Next, an overview of a process for creating a spherical image fromimages captured by the image capturing device 10 will be described withreference to FIGS. 3A to 4B. FIG. 3A is a view illustrating ahemispherical image (front side) captured by the image capturing device10, FIG. 3B is a view illustrating a hemispherical image (back side)captured by the image capturing device 10, and FIG. 3C is a viewillustrating an image in equirectangular projection (hereinafterreferred to as “equirectangular projection image”). FIG. 4A is aconceptual diagram illustrating how a surface of a sphere is coveredwith the equirectangular projection image, and FIG. 4B is a viewillustrating a spherical image.

As illustrated in FIG. 3A, an image obtained by the imaging element 103a is a curved hemispherical image (front side) captured through the lens102 a described below. Also, as illustrated in FIG. 3B, an imagecaptured by the imaging element 103 b is a curved hemispherical image(back side) captured through the lens 102 b described below. The imagecapturing device 10 combines the hemispherical image (front side) andthe hemispherical image (back side), which is flipped by 180 degrees, tocreate an equirectangular projection image EC illustrated in FIG. 3C.

Then, as illustrated in FIG. 4A, the image capturing device 10 uses OpenGraphics Library for Embedded Systems (OpenGL ES) to map theequirectangular projection image EC onto the surface of the sphere so asto cover the surface of the sphere to create a spherical image(spherical panoramic image) CE illustrated in FIG. 4B. In other words,the spherical image CE is represented such that the equirectangularprojection image EC corresponds to a surface facing the center of thesphere. OpenGL ES is a graphics library used for visualizingtwo-dimensional (2D) and three-dimensional (3D) data. The sphericalimage CE may be either a still image or a moving image.

As described above, the spherical image CE is an image mapped onto asphere surface so as to cover the sphere surface and may look strange tothe human eye. The image capturing device 10 displays the sphericalimage CE such that a predetermined area T included in the sphericalimage CE is represented as a flat image having fewer curves. Thus, it isless likely that a person viewing the spherical image CE feels strange.An image of the predetermined area T is hereinafter referred to as“predetermined-area image”. The display of the predetermined-area imagewill be described with reference to FIGS. 5 to 8.

FIG. 5 is a view illustrating positions of a virtual camera IC and thepredetermined area T in a case in which the spherical image CE is of athree-dimensional solid sphere. The virtual camera IC is at a positionof a point of view of a user who is viewing the spherical image CEdisplayed on the surface of the three-dimensional solid sphere. FIG. 6Ais a perspective view of the virtual camera IC and the predeterminedarea T illustrated in FIG. 5, and FIG. 6B is a view illustrating animage of the predetermined area T displayed on a display. In FIG. 6A,the spherical image CE illustrated in FIG. 5 is represented by athree-dimensional solid sphere CS. Assuming that the spherical image CEgenerated in the way described above is represented by the solid sphereCS, the virtual camera IC is located inside the spherical image CE, asillustrated in FIG. 5. The predetermined area T in the spherical imageCE is an imaging area of the virtual camera IC and is specified bypredetermined-area information indicating an imaging direction and anangle of view of the virtual camera IC in a three-dimensional virtualspace including the spherical image CE. Zooming of the predeterminedarea T may be represented by moving the virtual camera IC toward or awayfrom the spherical image CE. A predetermined-area image Q is an image ofthe predetermined area T in the spherical image CE. Accordingly, thepredetermined area T can be specified by an angle of view α and adistance f from the virtual camera IC to the spherical image CE (seeFIG. 7).

The predetermined-area image Q illustrated in FIG. 6A is displayed on apredetermined display as an image of the imaging area of the virtualcamera IC, as illustrated in FIG. 6B. The image illustrated in FIG. 6Bis a predetermined-area image represented by predetermined-areainformation that is set by default. In the following, a description willbe given using the imaging direction (ea, aa) and the angle of view (α)of the virtual camera IC. The predetermined area T may be indicated by,instead of the angle of view α and the distance f, the imaging area (X,Y, Z) of the virtual camera IC, which is the predetermined area T.

Next, a relationship between the predetermined-area information and theimage of the predetermined area T will be described with reference toFIG. 7. FIG. 7 is a view illustrating a relationship between thepredetermined-area information and the image of the predetermined areaT. As illustrated in FIG. 7, “ea” denotes an elevation angle, “aa”denotes an azimuth angle, and “a” denotes an angle of view of thevirtual camera IC. The position of the virtual camera IC is changed suchthat the point of gaze of the virtual camera IC, which is indicated bythe imaging direction (ea, aa), matches a central point CP(x, y) of thepredetermined area T serving as the imaging area of the virtual cameraIC. As illustrated in FIG. 7, the central point CP(x, y) of thepredetermined area T, whose diagonal angle of view is represented by theangle of view α of the virtual camera IC and is denoted by α, is used asa parameter (x, y) of the predetermined-area information. Thepredetermined-area image Q is an image of the predetermined area T inthe spherical image CE. The distance from the virtual camera IC to thecentral point CP(x, y) of the predetermined area T is denoted by “f”.The distance between the center point CP and a given vertex of thepredetermined area T is denoted by “L” (2L is a diagonal line). In FIG.7, a trigonometric function equation typically expressed by Equation (1)below is satisfied.

$\begin{matrix}{{L/f} = {\tan\left( {\alpha/2} \right)}} & (1)\end{matrix}$

The image capturing device 10 described above is an example of an imagecapturing device capable of acquiring a wide-angle view image, and thespherical image CE is an example of the wide-angle view image. Thewide-angle view image is typically an image captured using a wide-anglelens, such as a lens capable of capturing an image of a wider range thanthat the human eye can perceive. Further, the wide-angle view image istypically an image taken with a lens having a focal length of 35 mm orless in terms of 35 mm film.

FIG. 8 is a view illustrating a point in a three-dimensional Euclideanspace according to spherical coordinates. The position coordinates ofthe central point CP when expressed by the spherical polar coordinatesystem is defined as (r, θ, ϕ). The coordinates (r, θ, ϕ) represent theradial distance, the polar angle, and the azimuth angle, respectively.The radial distance r is a distance from the origin of athree-dimensional virtual space including the spherical image CE to thecentral point CP and is thus equal to the distance f. FIG. 8 illustratesa relationship among the radial distance r, the polar angle θ, and theazimuth angle ϕ. In the following, a description will be given using theposition coordinates (r, θ, ϕ) of the virtual camera IC.

Overview of Image Communication System

Next, an overview of an image communication system according to thisembodiment will be described with reference to FIG. 9. FIG. 9 is adiagram illustrating an example of the general arrangement of an imagecommunication system 1. The image communication system 1 illustrated inFIG. 9 is a system in which captured images such as video imagesdistributed from a plurality of distribution sites are displayed at aplurality of viewing sites to provide real-time viewing of wide-rangeimages (e.g., spherical images) obtained by capturing scenes at thedistribution sites. In this embodiment, the distribution site is alsoreferred to as “area”.

As illustrated in FIG. 9, the image communication system 1 includesimage capturing devices 10 (image capturing devices 10A and 10B) anddistribution terminals 30 (distribution terminals 30A and 30B) locatedat a plurality of distribution sites (distribution site A (area A) anddistribution site B (area B)), a communication management apparatus 50,and communication terminals 70 (communication terminals 70C and 70D)located at a plurality of viewing sites (viewing sites C and D). Theimage capturing devices 10A and 10B are hereinafter referred tocollectively as “image capturing devices 10” or individually as “imagecapturing device 10” unless distinguished. The distribution terminals30A and 30B are hereinafter referred to collectively as “distributionterminals 30” or individually as “distribution terminal 30” unlessdistinguished. The communication terminals 70C and 70D are hereinafterreferred to collectively as “communication terminals 70” or individuallyas “communication terminal 70” unless distinguished.

The distribution terminals 30, the communication management apparatus50, and the communication terminals 70 of the image communication system1 can communicate with each other via a communication network 100. Thecommunication network 100 is constructed by the Internet, a mobilecommunication network, a local area network (LAN), or the like. Thecommunication network 100 may include, in addition to a wiredcommunication network, a network based on a wireless communicationstandard such as third generation (3G), fourth generation (4G), fifthgeneration (5G), Wireless Fidelity (Wi-Fi) (Registered Trademark),Worldwide Interoperability for Microwave Access (WiMAX), or Long TermEvolution (LTE).

As described above, the image capturing device 10 is a special digitalcamera configured to capture an image of an object or surroundings suchas scenery to obtain two hemispherical images, from which a sphericalimage is generated. The captured image obtained by the image capturingdevice 10 may be a moving image or a still image or may include both amoving image and a still image. Further, the captured image may includean image and audio. The distribution terminal 30 is configured toacquire an image from the image capturing device 10 via a wired cablesuch as a Universal Serial Bus (USB) cable and distribute the acquiredimage to the communication terminal 70 via the communication managementapparatus 50. In one example, the image capturing device 10A and thedistribution terminal 30A are located at the same site, namely, thedistribution site A. The image capturing device 10B and the distributionterminal 30B are located at the same site, namely, the distribution siteB. The number of distribution sites used is not limited to two, and onedistribution site or three or more distribution sites may be used. Inaddition, the image capturing device 10 and the distribution terminal 30may be connected wirelessly using short-range wireless communication orthe like, instead of using a wired cable.

The communication management apparatus 50 controls communication betweenthe distribution terminals 30 and the communication terminals 70 andmanages types of image data (e.g., general image and special image) tobe transmitted and received. In one example, the special image is aspherical image. The communication management apparatus 50 is arrangedin a service company or the like that provides image communicationservices. The communication management apparatus 50 has a serverfunction and provides identification information of image data (imagedata ID), the IP address of an image capturing device, image typeinformation representing the type of the image data, and the like inresponse to a request from the communication terminal 70.

The communication management apparatus 50 may be constructed by a singlecomputer or a plurality of computers that are assigned to dividedcomponents (functions or means) as appropriate. All or some of thefunctions of the communication management apparatus 50 may beimplemented by a server computer existing in a cloud environment or aserver computer existing in an on-premise environment.

The communication terminal 70 is a computer such as a personal computer(PC), which is used by a user at each viewing site. The communicationterminal 70 displays an image (a still image and/or a moving image)distributed from the distribution terminal 30. The communicationterminal 70 acquires a spherical image, which is an image captured bythe image capturing device 10, via the communication network 100. Thecommunication terminal 70 has installed therein OpenGL ES and is capableof creating predetermined-area information indicating a partial area ofa spherical image sent from the distribution terminal or creating apredetermined-area image from the spherical image. In one example, thecommunication terminal 70C is placed at the viewing site C where a userC1 is located, and the communication terminal 70D is placed at theviewing site D where a user D1 is located.

The arrangement of the terminals and devices (i.e., the communicationterminals 70, the image capturing devices 10, and the distributionterminals 30) and the users C1 and D1 illustrated in FIG. 9 is anexample, and another example may be used. Each of the communicationterminals 70 is not limited to a PC and may be, for example, a tabletterminal, a smartphone, a wearable terminal, a projector (PJ), anInteractive White Board (IWB), which is an electronic whiteboard withmutual communication capability, a telepresence robot, or the like.

A distribution site in the image communication system 1 will now beschematically described with reference to FIG. 10. FIG. 10 is a viewillustrating an overview of a distribution site in the imagecommunication system 1. FIG. 10 illustrates the image capturing device10A and the distribution terminal 30A at the distribution site A. WhileFIG. 10 illustrates an example of the distribution site A, the sameapplies to other distribution sites such as the distribution site B.

The distribution site A illustrated in FIG. 10 is, for example, a spacesuch as an office floor, in which a plurality of image capturing devices10A (image capturing devices 10A-1 to 10A-8) and a distribution terminal30A capable of communicating with the image capturing devices 10A arearranged. The distribution site is not limited to the office floor andmay be any space for which a user (or viewer) at a viewing site desiresto remotely grasp the situation, and examples of the distribution siteinclude a school, a factory, a warehouse, a construction site, a serverroom, and a store. For example, the image capturing devices 10A arearranged at predetermined intervals on desks on the office floor. Theplurality of image capturing devices 10A are used to capture images ofthe entire distribution site A. The distribution terminal 30A receivescaptured image data from the image capturing devices 10A and distributesthe received captured image data to the communication terminals 70 atviewing sites. The numbers of image capturing devices 10A anddistribution terminals 30A arranged at the distribution site A are notlimited to those illustrated in FIG. 10. That is, as illustrated in FIG.9, one image capturing device 10A and one distribution terminal 30A maybe arranged at the distribution site A. As illustrated in FIG. 10, imagecapturing devices 10A and distribution terminals 30A having many-to-onerelationship may be arranged. The same applies to the distribution siteB.

In an existing system in which images captured by a plurality of imagecapturing devices arranged at a remote site are viewable at viewingsites, a user who desires to view the situation at a specific portion inthe distribution site does not know which of the images captured by theimage capturing devices to view. In addition, in a case in which usersat different viewing sites desire to view different portions in thedistribution site, operations on the image capturing devices (e.g.,pan-tilt-zoom (PTZ) operations) may conflict with each other. As aresult, it may be difficult to provide an exclusive viewing that allowsa plurality of users who view the same distribution site to viewdifferent portions. Further, even in a case in which spherical images,as described above, are captured by the image capturing devices, a userperforms individual operations on a plurality of spherical images toperform a process for displaying an intended portion, and it isdifficult for a user to perform an intuitive operation. To address thisinconvenience, the image communication system 1 allows a user to performan intuitive operation on a display screen such that a plurality ofusers are able to view, with interest, different portions in thedistribution site using a captured image acquired from the same imagecapturing device.

Hardware Configuration of Image Communication System

Next, the hardware configuration of the devices or terminals of theimage communication system 1 according to an embodiment will bedescribed with reference to FIGS. 11 and 12. In the hardwareconfigurations illustrated in FIGS. 11 and 12, certain hardware elementsmay be added or deleted as appropriate.

Hardware Configuration of Image Capturing Device

First, the hardware configuration of the image capturing device 10 willbe described with reference to FIG. 11. FIG. 11 is a diagramillustrating an example hardware configuration of the image capturingdevice 10. In the following description, the image capturing device 10is a spherical (omnidirectional) image capturing device including twoimaging elements. However, the image capturing device 10 may include anysuitable number of imaging elements greater than or equal to two imagingelements. In addition, the image capturing device 10 is not necessarilya device dedicated to capturing of an omnidirectional image.Alternatively, a typical digital camera, smartphone, or the like may beequipped with an external omnidirectional image capturing unit toimplement an image capturing device having substantially the samefunctions as those of the image capturing device 10.

As illustrated in FIG. 11, the image capturing device 10 includes animaging device 101, an image processor 104, an imaging controller 105, amicrophone 108, an audio processor 109, a central processing unit (CPU)111, a read only memory (ROM) 112, a static random access memory (SRAM)113, a dynamic random access memory (DRAM) 114, the operation device115, an input/output interface (I/F) 116, a short-range communicationcircuit 117, an antenna 117 a of the short-range communication circuit117, an electronic compass 118, a gyro sensor 119, an accelerationsensor 120, and a network I/F 121.

The imaging device 101 includes wide-angle lenses (so-called fish-eyelenses) 102 a and 102 b, each having an angle of view greater than orequal to 180 degrees so as to form a hemispherical image. The imagingdevice 101 further includes the two imaging elements 103 a and 103 bcorresponding to the wide-angle lenses 102 a and 102 b, respectively.The imaging elements 103 a and 103 b each include an image sensor suchas a complementary metal oxide semiconductor (CMOS) sensor or acharge-coupled device (CCD) sensor, a timing generation circuit, and agroup of registers. Each of the image sensors converts an optical imageformed by a corresponding one of the wide-angle lenses 102 a and 102 binto an electric signal to output image data. Each of the timinggeneration circuits generates a horizontal or vertical synchronizationsignal, a pixel clock, and the like for a corresponding one of the imagesensors. Each of the groups of registers has set therein variouscommands, parameters, and the like to be used for operations of acorresponding one of the imaging elements 103 a and 103 b.

The imaging elements 103 a and 103 b of the imaging device 101 areconnected to the image processor 104 via respective parallel I/F buses.In addition, the imaging elements 103 a and 103 b of the imaging device101 are connected to the imaging controller 105 via respective serialI/F buses such as inter-integrated circuit (I2C) buses. The imageprocessor 104, the imaging controller 105, and the audio processor 109are connected to the CPU 111 via a bus 110. The ROM 112, the SRAM 113,the DRAM 114, the operation device 115, the input/output I/F 116, theshort-range communication circuit 117, the electronic compass 118, thegyro sensor 119, the acceleration sensor 120, and the network I/F 121are also connected to the bus 110.

The image processor 104 acquires respective pieces of image data outputfrom the imaging elements 103 a and 103 b via the parallel I/F buses andperforms predetermined processing on the pieces of image data.Thereafter, the image processor 104 combines the pieces of image data,which are subjected to the predetermined processing, to generate data ofan equirectangular projection image as illustrated in FIG. 3C.

The imaging controller 105 usually functions as a master device whilethe imaging elements 103 a and 103 b usually function as slave devices.The imaging controller 105 sets commands and the like in the groups ofregisters of the imaging elements 103 a and 103 b via the respective I2Cbuses. The imaging controller 105 receives various commands from the CPU111. Further, the imaging controller 105 acquires status data and thelike of the groups of registers of the imaging elements 103 a and 103 bvia the respective I2C buses. The imaging controller 105 sends theacquired status data and the like to the CPU 111.

The imaging controller 105 instructs the imaging elements 103 a and 103b to output image data at a time when a shutter button of the operationdevice 115 is pressed (or tapped). Such a pressing or tapping operationis hereinafter referred to simply as “operation”. In some cases, theimage capturing device 10 has a function of displaying a preview imageon a display (e.g., a display of an external terminal such as asmartphone that performs short-range communication with the imagecapturing device 10 through the short-range communication circuit 117)or displaying a moving image. In the case of displaying a moving image,the imaging elements 103 a and 103 b continuously output image data at apredetermined frame rate (frames per minute).

Further, as described below, the imaging controller 105 operates incooperation with the CPU 111 to also function as a synchronizationcontroller for synchronizing the time when the imaging element 103 aoutputs image data and the time when the imaging element 103 b outputsimage data. Although the image capturing device 10 does not include adisplay in this embodiment, the image capturing device 10 may include adisplay. The microphone 108 converts sounds into audio data (signal).The audio processor 109 acquires the audio data output from themicrophone 108 via an I/F bus and performs predetermined processing onthe audio data.

The CPU 111 controls the entire operation of the image capturing device10 and also performs certain processing. The ROM 112 stores variousprograms for the CPU 111. The SRAM 113 and the DRAM 114 each operate asa work memory to store programs to be executed by the CPU 111 or databeing processed. More specifically, the DRAM 114 stores image data beingprocessed by the image processor 104 or data of the equirectangularprojection image on which processing has been performed.

The operation device 115 generally refers to various operation keys, apower switch, a shutter button, a touch panel having both the displayand operation functions, and the like. The user operates the operationdevice 115 to input various image capturing modes, image capturingconditions, or the like.

The input/output I/F 116 generally refers to an interface circuit suchas a USB I/F that allows the image capturing device 10 to communicatewith an external medium such as a Secure Digital (SD) card or a personalcomputer. The input/output I/F 116 may be either wired or wireless. Thedata of the equirectangular projection image, which is stored in theDRAM 114, is stored in an external medium via the input/output I/F 116or transmitted to an external terminal (or apparatus) via theinput/output I/F 116, as appropriate.

The short-range communication circuit 117 communicates with an externalterminal (or apparatus) via the antenna 117 a of the image capturingdevice 10 using short-range wireless communication technology such asnear-field communication (NFC), Bluetooth (registered trademark), orWi-Fi (registered trademark). The short-range communication circuit 117is capable of transmitting the data of the equirectangular projectionimage to an external terminal (or apparatus).

The electronic compass 118 calculates an orientation of the imagecapturing device from the Earth's magnetism and outputs orientationinformation. The orientation information is an example of relatedinformation (metadata) in compliance with Exchangeable Image File Format(EXIF) and is used for image processing such as image correction ofcaptured images. The related information also includes data such as thedate and time when the image is captured, and the data size of the imagedata. The gyro sensor 119 detects a change in angle of the imagecapturing device 10 (roll angle, pitch angle, and yaw angle) withmovement of the image capturing device 10. The change in angle is anexample of related information (metadata) in compliance with EXIF and isused for image processing such as image correction of captured images.The acceleration sensor 120 detects acceleration in three axialdirections. The image capturing device 10 calculates the position (anangle with respect to the direction of gravity) of the image capturingdevice 10, based on the acceleration detected by the acceleration sensor120. The image capturing device 10 provided with the acceleration sensor120 improves the accuracy of image correction. The network I/F 121 is aninterface for performing data communication using the communicationnetwork 100 such as the Internet.

Hardware Configuration of Distribution Terminal

FIG. 12 is a diagram illustrating an example hardware configuration ofthe distribution terminal 30. The hardware components of thedistribution terminal 30 are denoted by reference numerals in the 300 s.The distribution terminal 30 is constructed by a computer. Asillustrated in FIG. 12, the distribution terminal 30 includes a CPU 301,a ROM 302, a random access memory (RAM) 303, a hard disk (HD) 304, ahard disk drive (HDD) controller 305, a display 306, an external deviceconnection I/F 308, a network I/F 309, a bus line 310, a keyboard 311, apointing device 312, a digital versatile disk rewritable (DVD-RW) drive314, a media I/F 316, an audio input/output I/F 317, a microphone 318, aspeaker 319, and a short-range communication circuit 320.

The CPU 301 controls the entire operation of the distribution terminal30. The ROM 302 stores a program used for driving the CPU 301, such asan initial program loader (IPL). The RAM 303 is used as a work area forthe CPU 301. The HD 304 stores various data such as a program. The HDDcontroller 305 controls reading or writing of various data from or tothe HD 304 under the control of the CPU 301. The display 306 displaysvarious kinds of information such as a cursor, a menu, a window,characters, and an image. The display 306 is an example of a displaydevice. In one example, the display 306 is a touch panel displayprovided with an input device. The external device connection I/F 308 isan interface for connecting to various external devices. Examples of theexternal devices include, but are not limited to, a USB memory and aprinter. The network I/F 309 is an interface for performing datacommunication using the communication network 100. The bus line 310 isan address bus, a data bus, or the like for electrically connecting thehardware elements illustrated in FIG. 12, such as the CPU 301.

The keyboard 311 is a type of input device provided with a plurality ofkeys for inputting characters, numerals, various instructions, or thelike. The pointing device 312 is a type of input device for selecting orexecuting various instructions, selecting a processing target, or movinga cursor being displayed. The input device is not limited to thekeyboard 311 and the pointing device 312 and may be a touch panel, avoice input device, or the like. The DVD-RW drive 314 controls readingor writing of various data from or to a DVD-RW 313, which is an exampleof a removable recording medium. The removable recording medium is notlimited to the DVD-RW and may be a digital versatile disk recordable(DVD-R), a Blu-ray Disc (registered trademark), or the like. The mediaI/F 316 controls reading or writing (storing) of data from or to arecording medium 315 such as a flash memory. The microphone 318 is atype of built-in sound collector for receiving input sounds. The audioinput/output I/F 317 is a circuit that processes input and output of anaudio signal between the microphone 318 and the speaker 319 under thecontrol of the CPU 301. The short-range communication circuit 320 is acommunication circuit for performing communication with an externalterminal (or apparatus) using short-range wireless communicationtechnology such as NFC, Bluetooth, or Wi-Fi.

Hardware Configuration of Communication Management Apparatus

FIG. 12 is a diagram illustrating an example hardware configuration ofthe communication management apparatus 50. The hardware components ofthe communication management apparatus 50 are denoted by referencenumerals in the 500 s in parentheses. The communication managementapparatus 50 is constructed by a computer. As illustrated in FIG. 12,since the communication management apparatus 50 has a configurationsimilar to that of the distribution terminal 30, the description of thehardware components will be omitted.

Hardware Configuration of Communication Terminal

FIG. 12 is a diagram illustrating an example hardware configuration ofthe communication terminal 70. The hardware components of thecommunication terminal 70 are denoted by reference numerals in the 700 sin parentheses. The communication terminal 70 is constructed by acomputer. As illustrated in FIG. 12, since the communication terminal 70has a configuration similar to that of the distribution terminal 30, thedescription of the hardware components will be omitted.

Further, each of the programs described above may be recorded in a filein an installable or executable format on a computer-readable recordingmedium for distribution. Examples of the recording medium include acompact disc recordable (CD-R), a digital versatile disk (DVD), aBlu-ray Disc, an SD card, and a USB memory. In addition, the recordingmedium may be provided in the form of a program product to users withina certain country or outside that country. For example, in thecommunication management apparatus 50, a program according to anembodiment of the present disclosure is executed to implement acommunication management method according to an embodiment of thepresent disclosure.

Functional Configuration of Image Communication System

Next, the functional configuration of the image communication system 1according to an embodiment will be described with reference to FIGS. 13to 20B. FIGS. 13 and 14 are diagrams illustrating an example functionalconfiguration of the image communication system 1. FIGS. 13 and 14illustrate devices and terminals related to the processes or operationsdescribed below among the devices and terminals illustrated in FIG. 9.

Functional Configuration of Image Capturing Device

First, the functional configuration of the image capturing device 10will be described with reference to FIG. 13. The image capturing device10 includes a communication unit 11, an acceptance unit 12, an imagingunit 13, a sound collection unit 14, and a storing and reading unit 19.The communication unit 11, the acceptance unit 12, the imaging unit 13,the sound collection unit 14, and the storing and reading unit 19 arefunctions or means implemented by any one of the hardware elementsillustrated in FIG. 11 operating in accordance with instructions fromthe CPU 111 according to an image capturing device program loaded ontothe DRAM 114 from the SRAM 113. The image capturing device 10 furtherincludes a storage unit 1000. The storage unit 1000 is constructed bythe ROM 112, the SRAM 113, and the DRAM 114 illustrated in FIG. 11. Thestorage unit 1000 stores the globally unique identifier (GUID) of theimage capturing device 10.

The communication unit 11 is mainly implemented by processing performedby the CPU 111 and communicates various data or information to anotherapparatus or terminal. The communication unit 11 performs, for example,data communication with another apparatus or terminal through theshort-range communication circuit 117 using short-range wirelesscommunication technology. Further, the communication unit 11 performs,for example, data communication with another apparatus or terminalthrough the input/output I/F 116 via various cables or the like. Thecommunication unit 11 further performs data communication with anotherapparatus or terminal through the network I/F 121 via the communicationnetwork 100.

The acceptance unit 12 is mainly implemented by processing performed bythe CPU 111 on the operation device 115 and accepts various selectionsor inputs from a user. The imaging unit 13 is mainly implemented byprocessing performed by the CPU 111 on the imaging device 101, the imageprocessor 104, and the imaging controller 105 and captures an objectsuch as scenery to acquire captured image data. The sound collectionunit 14 is mainly implemented by processing performed by the CPU 111 onthe microphone 108 and the audio processor 109 and collects soundsaround the image capturing device 10.

The storing and reading unit 19 is mainly implemented by processingperformed by the CPU 111 and stores various data (or information) in thestorage unit 1000 or reads various data (or information) from thestorage unit 1000.

Functional Configuration of Distribution Terminal

Next, the functional configuration of the distribution terminal 30 willbe described with reference to FIG. 13. The distribution terminal 30includes a transmitting/receiving unit 31, an acceptance unit 32, animage and audio processing unit 33, a display control unit 34, adetermination unit 35, a creation unit 36, a communication unit 37, anda storing and reading unit 39. The transmitting/receiving unit 31, theacceptance unit 32, the image and audio processing unit 33, the displaycontrol unit 34, the determination unit 35, the creation unit 36, thecommunication unit 37, and the storing and reading unit 39 are functionsor means implemented by any one of the hardware elements illustrated inFIG. 12 operating in accordance with instructions from the CPU 301according to a distribution terminal program loaded onto the RAM 303from the HD 304. The distribution terminal 30 further includes a storageunit 3000. The storage unit 3000 is constructed by the ROM 302, the RAM303, and the HD 304 illustrated in FIG. 12.

The transmitting/receiving unit 31 is mainly implemented by processingperformed by the CPU 301 on the network I/F 309 and transmits andreceives various data or information to and from another apparatus orterminal via the communication network 100.

The acceptance unit 32 is mainly implemented by processing performed bythe CPU 301 on the keyboard 311 or the pointing device 312 and acceptsvarious selections or inputs from a user.

The image and audio processing unit 33 is mainly implemented byprocessing performed by the CPU 301 and performs image processing oncaptured image data acquired by the image capturing device 10 capturingan object. The image and audio processing unit 33 further performs audioprocessing on audio data of a voice signal, which is obtained byconverting the voice of the user using the microphone 318. For example,the image and audio processing unit 33 performs image processing oncaptured image data received from the image capturing device 10, basedon image type information such as the source name so that the displaycontrol unit 34 causes the display 306 to display an image.Specifically, when the image type information indicates the specialimage, the image and audio processing unit 33 converts the capturedimage data (e.g., data of hemispherical images as illustrated in FIGS.3A and 3B) into spherical image data as illustrated in FIG. 4B to createspherical image data. Further, the image and audio processing unit 33outputs a voice signal of audio data distributed from another terminalvia the communication management apparatus 50 to the speaker 319 andoutputs a voice from the speaker 319.

The display control unit 34 is mainly implemented by processingperformed by the CPU 301 and causes the display 306 to display variousimages, characters, or the like. The determination unit 35 isimplemented by processing performed by the CPU 301 and performs variousdeterminations. For example, the determination unit 35 determines theimage type of captured image data received from the image capturingdevice 10.

The creation unit 36 is mainly implemented by processing performed bythe CPU 301 and creates a source name, which is an example of the imagetype information, in accordance with a naming rule, based on the generalimage or special image (that is, the spherical image) determined by thedetermination unit 35. For example, if the determination unit 35determines that the image type is the general image, the creation unit36 creates the source name “Video” indicating the general image. Bycontrast, if the determination unit 35 determines that the image type isthe special image, the creation unit 36 creates the source name“Video_Omni” indicating the special image.

The communication unit 37 is mainly implemented by processing performedby the CPU 301 on the short-range communication circuit 320 andcommunicates with the communication unit 11 of the image capturingdevice 10 using short-range wireless communication technology such asNFC, Bluetooth, or WiFi. In the foregoing description, the communicationunit 37 and the transmitting/receiving unit 31 are configured asseparate communication units. In another example, the communication unit37 and the transmitting/receiving unit 31 may share a singlecommunication unit.

The storing and reading unit 39 is mainly implemented by processingperformed by the CPU 301 and stores various data (or information) in thestorage unit 3000 or reads various data (or information) from thestorage unit 3000.

Image Capturing Device Management Table

FIG. 15A is a conceptual diagram illustrating an example image capturingdevice management table. The storage unit 3000 includes an imagecapturing device management database (DB) 3001. The image capturingdevice management DB 3001 is implemented by an image capturing devicemanagement table illustrated in FIG. 15A. The image capturing devicemanagement table stores and manages a vendor ID and a product ID in theGUID of an image capturing device capable of obtaining two hemisphericalimages from which a spherical image is generated. Examples of the GUIDinclude a vendor ID (VID) and a product ID (PID), which are used by aUSB device. The vendor ID and the product ID may be stored when thedistribution terminal 30 is shipped from the factory or may beadditionally stored after the distribution terminal 30 is shipped fromthe factory, for example.

Image Type Management Table

FIG. 15B is a conceptual diagram illustrating an example image typemanagement table. The storage unit 3000 includes an image typemanagement DB 3002. The image type management DB 3002 is implemented byan image type management table illustrated in FIG. 15B. The image typemanagement table stores, for each image data ID, an Internet Protocol(IP) address of an image capturing device, which is an example of anaddress of an image capturing device, and a source name (image typeinformation) in association with each other. The image data ID is anexample of image data identification information for identifying imagedata of an image to be distributed. The IP address of the imagecapturing device indicates the IP address of the image capturing device10 that has captured the image data indicated by the associated imagedata ID. The source name is a name for specifying the image capturingdevice 10 that has captured the image data indicated by the associatedimage data ID, and is an example of image type information. The sourcename is a name created by the distribution terminal 30 in accordancewith a predetermined naming rule.

The illustrated example indicates that four image capturing deviceshaving the IP addresses “2.1.2.3”, “2.1.1.5”, “2.1.5.4”, and “2.1.5.6”have transmitted image data indicated by image data IDs “RS001”,“RS002”, “RS003”, and “RS004”, respectively. It is also indicated thatthe image types indicated by the source names of the four imagecapturing devices are “Video_Omni”, “Video_Omni”, “Video”, and “Video”,which indicate the image types “special image”, “special image”,“general image”, and “general image”, respectively. In this embodiment,the special image is the spherical image. The IP address is an exampleof address information, and the address information may be a MediaAccess Control (MAC) address, a terminal identification (ID), or thelike. While the IP address is a simplified representation of theInternet Protocol version 4 (IPv4) address, the FP address may be anInternet Protocol version 6 (IPv6) address. In addition, data other thanimage data may be managed in association with the image data ID.Examples of the data other than image data include audio data, anddocument data to be used to share the screen between the distributionsite and the viewing sites.

Functional Configuration of Communication Management Apparatus

Next, the functional configuration of the communication managementapparatus 50 will be described with reference to FIG. 14. Thecommunication management apparatus 50 includes a transmitting/receivingunit 51, a selection unit 53, a determination unit 55, a generation unit56, a distribution site management unit 57, and a storing and readingunit 59. The transmitting/receiving unit 51, the selection unit 53, thedetermination unit 55, the generation unit 56, the distribution sitemanagement unit 57, and the storing and reading unit 59 are functions ormeans implemented by any one of the hardware elements illustrated inFIG. 12 operating in accordance with instructions from the CPU 501according to a communication management apparatus program loaded ontothe RAM 503 from the HD 504. The communication management apparatus 50further includes a storage unit 5000. The storage unit 5000 isconstructed by the ROM 502, the RAM 503, and the HD 504 illustrated inFIG. 12.

The transmitting/receiving unit 51 is mainly implemented by processingperformed by the CPU 501 on the network I/F 509 and transmits andreceives various data or information to and from another apparatus viathe communication network 100.

The selection unit 53 is mainly implemented by processing performed bythe CPU 501 and selects an image capturing device 10 in response to arequest from the communication terminal 70. Specifically, the selectionunit 53 selects a specific image capturing device 10 among the pluralityof image capturing devices 10, based on, for example, coordinateinformation of a point of interest P, which is accepted by thecommunication terminal 70, arrangement position information of theplurality of image capturing devices 10, and number-of-divisioninformation indicating the number of divisions of an area accepted bythe communication terminal 70. The selected specific image capturingdevice 10 is referred to as “supplementary image capturing device”, inparticular. In this embodiment, the supplementary image capturing devicefunctions as an example of a second image capturing device. In thisembodiment, the selection unit 53 functions as an example of selectionmeans.

The determination unit 55 is mainly implemented by processing performedby the CPU 501 and performs various determinations.

The generation unit 56 is mainly implemented by processing performed bythe CPU 501 and generates an image data ID and predetermined-areainformation. The generation unit 56 generates, for example,predetermined-area information. The predetermined-area informationindicates a predetermined area (e.g., the predetermined area Tillustrated in FIG. and the like) in an image captured by the imagecapturing device 10 selected by the selection unit 53. An image in whichthe entire captured image is displayed (e.g., the spherical image CEillustrated in FIG. 5 and the like) is also referred to as “entireimage”. The distribution site management unit 57 is mainly implementedby processing performed by the CPU 501 and manages distribution siteinformation indicating the state of the distribution site.

The storing and reading unit 59 is mainly implemented by processingperformed by the CPU 501 and stores various data (or information) in thestorage unit 5000 or reads various data (or information) from thestorage unit 5000.

Session Management Table

FIG. 16A is a conceptual diagram illustrating an example sessionmanagement table. The storage unit 5000 includes a session management DB5001. The session management DB 5001 is implemented by a sessionmanagement table illustrated in FIG. 16A. The session management tablestores, for each session ID, a site ID and an IP address of aparticipant communication terminal in association with each other. Thesession ID is an example of session identification information foridentifying a communication session for implementing imagecommunication. The session ID is generated for each virtual floor. Thesession ID is also managed by the communication terminals 70 and is usedwhen each of the communication terminals 70 selects a communicationsession. The site ID is an example of site identification informationfor identifying a distribution site. The IP address of the participantcommunication terminal indicates the IP address of a communicationterminal 70 participating in a virtual floor indicated by the associatedsession ID.

Image Type Management Table

FIG. 16B is a conceptual diagram illustrating an example image typemanagement table. The storage unit 5000 includes an image typemanagement DB 5002. The image type management DB 5002 is implemented byan image type management table illustrated in FIG. 16B. The image typemanagement table stores the information managed by the image typemanagement table illustrated in FIG. 15B and the same session ID as thesession ID managed in the session management table illustrated in FIG.16A in association with each other. The communication managementapparatus 50 stores an image data ID, an IP address of an imagecapturing device, and image type information, which are the same asthose managed in the distribution terminal 30 and the communicationterminal 70, because, for example, when a new communication terminal 70enters a virtual floor, the communication management apparatus 50transmits information including the image type information to acommunication terminal 70 that is already in video communication and thenew communication terminal 70, which has newly participated in the videocommunication. As a result, the communication terminal 70 that isalready in the video communication and the communication terminal 70that has newly participated in the video communication do not have totransmit and receive such information including the image typeinformation.

Predetermined-Area Management Table

FIG. 17A is a conceptual diagram illustrating an examplepredetermined-area management table. The storage unit 5000 includes apredetermined-area management DB 5003. The predetermined-area managementDB 5003 is implemented by a predetermined-area management tableillustrated in FIG. 17A. The predetermined-area management table stores,for the IP address of each image capturing device (image source) fromwhich captured image data is transmitted, an IP address of acommunication terminal (image destination) to which the captured imagedata is transmitted, and predetermined-area information in associationwith each other. The predetermined-area information indicates aparameter of a predetermined-area image currently displayed on thecommunication terminal to which the captured image data is transmitted.As illustrated in FIGS. 6A, 6B, and 7, the predetermined-areainformation is a conversion parameter for converting the captured imageinto the image of the predetermined area T (the predetermined-area imageQ) in the captured image.

For example, the information managed in the first row of thepredetermined-area management table illustrated in FIG. 17A indicatesthat after the distribution terminal 30 receives captured image dataobtained by the image capturing device 10 having the IP address“2.1.2.3”, the captured image data is transmitted from the distributionterminal 30 to the communication terminal 70 having the IP address“1.2.1.3” via the communication management apparatus 50. It is alsomanaged that the distribution terminal 30 is a distribution terminalfrom which the four pieces of predetermined-area information in thefirst to fourth rows of the predetermined-area management tableillustrated in FIG. 17A are transmitted.

When predetermined-area information including IP addresses in the sameset as an already managed set of the IP address of an image capturingdevice from which captured image data is transmitted and the IP addressof a communication terminal to which the captured image data istransmitted is newly received by the transmitting/receiving unit 51, thestoring and reading unit 59 rewrites the already managedpredetermined-area information to the newly received predetermined-areainformation.

Arrangement Information Management Table

FIG. 17B is a conceptual diagram illustrating an example arrangementinformation management table. The storage unit 5000 includes anarrangement information management DB 5004. The arrangement informationmanagement DB 5004 is implemented by an arrangement informationmanagement table illustrated in FIG. 17B. The arrangement informationmanagement table stores, for each site ID for identifying a distributionsite, an IP address of an image capturing device 10 arranged in thedistribution site, coordinate values indicating the position at whichthe image capturing device 10 is arranged, and the arrangement directionof the image capturing device 10 in association with each other. Thecoordinate values are coordinate values on the map of the distributionsite where the image capturing device 10 is arranged. The arrangementdirection of the image capturing device 10 indicates the direction(angle) on the map in which the front surface of the image capturingdevice 10 faces. The arrangement direction is set to a desired direction(angle) by the administrator of the distribution terminal 30 or thedistribution site.

Distribution-Site Management Table

FIG. 18 is a conceptual diagram illustrating an exampledistribution-site management table. The storage unit 5000 includes adistribution site management DB 5005. The distribution site managementDB 5005 is implemented by a distribution-site management tableillustrated in FIG. 18. The distribution-site management table stores,for each site ID for identifying a distribution site, a site name anddistribution site information in association with each other. Thedistribution site information indicates the state of the distributionsite. The distribution site information includes a uniform resourcelocator (URL) for accessing map image data indicating a map of thedistribution site and site coordinate information indicating thecoordinate values of the distribution site. The communication terminal70 accesses the URL transmitted from the communication managementapparatus 50 to acquire the map image data and the site coordinateinformation of the distribution site. The URL is an example of storagedestination information. The storage destination information is notlimited to the URL and may be a uniform resource identifier (URI) or thelike.

Weighting Coefficient Management Table

FIG. 19A is a conceptual diagram illustrating an example weightingcoefficient management table. The storage unit 5000 includes a weightingcoefficient management DB 5006. The weighting coefficient management DB5006 is implemented by a weighting coefficient management tableillustrated in FIG. 19A. The weighting coefficient management tablestores, for each reference image capturing device name indicating thename of a reference image capturing device, a coefficient and asupplementary image capturing device name, which indicates the name of asupplementary image capturing device, in association with each other.The reference image capturing device is an image capturing device 10arranged at the position closest to a point of interest P (predeterminedposition) designated by a user. The supplementary image capturing deviceis an image capturing device 10 capable of capturing from the mostdifferent aspects a portion of the point of interest P for which animage is difficult for the reference image capturing device to capture.The coefficient is a value indicating the arrangement relationshipbetween the reference image capturing device and the supplementary imagecapturing device.

The coefficient is set to a desired value in advance by theadministrator who arranges the image capturing devices 10 at thedistribution site A and the like or the administrator who manages thecommunication management apparatus 50. For example, for some pairs ofsupplementary image capturing device and the reference image capturingdevice, a coefficient of 10.0 is set due to presence of walltherebetween (specifically, CAM1 and CAM3, CAM 2 and CAM 3). In anotherexample, for some pairs of supplementary image capturing device and thereference image capturing device, a coefficient of 5.0 is set due topresence of a partial wall therebetween.

Specific Weighting Coefficient Management Table

FIG. 19B is a conceptual diagram illustrating an example specificweighting coefficient management table. The storage unit 5000 includes aspecific weighting coefficient management DB 5007. The specificweighting coefficient management DB 5007 is implemented by a specificweighting coefficient management table illustrated in FIG. 19B. Thespecific weighting coefficient management table stores, for eachreference image capturing device name indicating the name of a referenceimage capturing device, a specific coefficient and a supplementary imagecapturing device name, which indicates the name of a supplementary imagecapturing device, in association with each other. The specificcoefficient is given as a value larger than any other coefficient in theweighting coefficient management table illustrated in FIG. 19A when atleast one of the following conditions is satisfied: (1) an obstacle(such as a wall) is present between the reference image capturing deviceand the supplementary image capturing device; and (2) the supplementaryimage capturing device has any abnormality or the like. The specificcoefficient is also set to a desired value in advance by theadministrator who arranges the image capturing devices 10 at thedistribution site A and the like or the administrator who manages thecommunication management apparatus 50.

Functional Configuration of Communication Terminal

Next, the functional configuration of the communication terminal 70 willbe described with reference to FIG. 14. The communication terminal 70includes a transmitting/receiving unit (acquisition unit) 71, anacceptance unit 72, an image and audio processing unit 73, a displaycontrol unit 74, a determination unit 75, a creation unit 76, apoint-of-interest specifying unit 77, a selection unit 81, a generationunit 82, and a storing and reading unit 79. The transmitting/receivingunit 71, the acceptance unit 72, the image and audio processing unit 73,the display control unit 74, the determination unit 75, the creationunit 76, the point-of-interest specifying unit 77, the selection unit81, the generation unit 82, and the storing and reading unit 79 arefunctions or means implemented by any one of the hardware elementsillustrated in FIG. 12 operating in accordance with instructions fromthe CPU 701 according to a communication terminal program loaded ontothe RAM 703 from the HD 704. The communication terminal 70 furtherincludes a storage unit 7000. The storage unit 7000 is constructed bythe ROM 702, the RAM 703, and the HD 704 illustrated in FIG. 12.

The transmitting/receiving unit (acquisition unit) 71 is mainlyimplemented by processing performed by the CPU 701 on the network I/F709 and transmits and receives various data or information to and fromanother apparatus or terminal via the communication network 100. Thetransmitting/receiving unit (acquisition unit) 71 receives, for example,captured image data distributed from the distribution terminal 30 viathe communication management apparatus 50. Further, thetransmitting/receiving unit (acquisition unit) 71 functions as, forexample, an acquisition unit and acquires predetermined-areainformation. The predetermined-area information indicates apredetermined area including the point of interest P in an imagecaptured by a specific image capturing device 10 selected based on thecoordinate information (predetermined-position information) of the pointof interest P, which is accepted by the acceptance unit 72, and thearrangement positions (arrangement position information) of the imagecapturing devices 10.

The acceptance unit 72 is mainly implemented by processing performed bythe CPU 701 on the keyboard 711 or the pointing device 712 and acceptsvarious selections or inputs from a user. For example, the acceptanceunit 72 accepts input of the point of interest P, which is designated bythe user, in a distribution site and the number of divisions into whicha predetermined range (e.g., a circle having a radius r) centered on thepoint of interest P is divided from the point of interest P. In thisembodiment, the acceptance unit 72 functions as, or has means forfunctioning as, an example of acceptance means.

The image and audio processing unit 73 is mainly implemented byprocessing performed by the CPU 701 and performs image processing oncaptured image data distributed from the distribution terminal 30. Theimage and audio processing unit 73 further performs audio processing onaudio data distributed from the distribution terminal 30. For example,to display on the display 706 an image of a predetermined areacorresponding to predetermined-area information received by thetransmitting/receiving unit (acquisition unit) 71, the image and audioprocessing unit 73 applies perspective projection conversion to thecaptured image (spherical image) using the predetermined-areainformation to generate a predetermined-area image corresponding to thepredetermined-area information. Further, the image and audio processingunit 73 outputs a voice signal of audio data distributed from thedistribution terminal 30 via the communication management apparatus 50to the speaker 719 and outputs a voice from the speaker 719.

The display control unit 74 is mainly implemented by processingperformed by the CPU 701 and causes the display 706 to display variousimages, characters, or the like. For example, the display control unit74 causes the display 706 to display the predetermined-area imagegenerated by the image and audio processing unit 73.

The determination unit 75 is implemented by processing performed by theCPU 701 and performs various determinations. In this embodiment, thedisplay control unit 74 functions as, or has means for functioning as,an example of display control means.

The creation unit 76 is mainly implemented by processing performed bythe CPU 701 and implements functions similar to those of the creationunit 36. The point-of-interest specifying unit 77 is mainly implementedby processing performed by the CPU 701 and specifies the point ofinterest P in the distribution site where the image capturing devices 10are arranged. For example, the point-of-interest specifying unit 77specifies the coordinate information of the point of interest Pdesignated by the user on the map image of the distribution site.

The selection unit 81 is mainly implemented by processing performed bythe CPU 701 and implements functions similar to those of the selectionunit 53. The generation unit 82 is mainly implemented by processingperformed by the CPU 701 and generates predetermined-area information.For example, the generation unit 82 generates predetermined-areainformation indicating a predetermined area in an image captured by animage capturing device 10 selected by the selection unit 81. Further,the generation unit 82 functions as, for example, an acquisition unitand acquires predetermined-area information. The predetermined-areainformation indicates a predetermined area including the point ofinterest P in an image captured by a specific image capturing device 10selected based on the coordinate information of the point of interest P,which is accepted by the acceptance unit 72, and the arrangementpositions of the image capturing devices 10.

The storing and reading unit 79 is mainly implemented by processingperformed by the CPU 701 and stores various data (or information) in thestorage unit 7000 or reads various data (or information) from thestorage unit 7000.

Image Type Management Table

FIG. 20A is a conceptual diagram illustrating an example image typemanagement table. The storage unit 7000 includes an image typemanagement DB 7001. The image type management DB 7001 is implemented byan image type management table illustrated in FIG. 20A. Since the imagetype management table has substantially the same data configuration asthat of the image type management table illustrated in FIG. 15B, thedescription thereof will be omitted.

Predetermined-Area Management Table

FIG. 20B is a conceptual diagram illustrating an examplepredetermined-area management table. The storage unit 7000 includes apredetermined-area management DB 7002. The predetermined-area managementDB 7002 is implemented by a predetermined-area management tableillustrated in FIG. 20B. The predetermined-area management table stores,for the IP address of each image capturing device that has obtainedcaptured image data, predetermined-area information in association witheach other. The predetermined-area information indicates a currentlydisplayed predetermined-area image. As illustrated in FIGS. 6A, 6B, and7, the predetermined-area information is a conversion parameter forconverting the captured image into the image of the predetermined area T(the predetermined-area image Q) in the captured image.

The relationship between the point of interest P and a plurality ofimage capturing devices 10 arranged at the distribution site A will nowbe described with reference to FIGS. 21 to 27.

Relationship Between Point of Interest P and Image Capturing Deviceswhen Number of Divisions is Two

FIG. 21 is a view schematically describing an example relationshipbetween the point of interest P and the image capturing devices 10 whenthe number of divisions is two. For example, at the distribution site A(area A) in the image communication system 1 illustrated in FIG. 10, theplurality of image capturing devices 10A are arranged regularly in agrid pattern, as illustrated in FIG. 21. Based on the arrangement of theplurality of image capturing devices 10A, the generation unit 56generates a two-dimensional arrangement image (map image) based on thecoordinate values of each of the image capturing devices 10A, forexample. At this time, the user who uses the communication terminal 70operates a “point of interest” button 605 and a “number of divisions”button 670 (see FIG. 34). The “point of interest” button 605 indicates apredetermined position. The “number of divisions” button 670 is used todivide a predetermined range (e.g., a circle having a radius r) centeredon the point of interest P from the point of interest P. The user canoperate the “point of interest” button 605 while referring to aschematic image 650 corresponding to a two-dimensional arrangement imagedisplayed on a site display screen 600 of the communication terminal 70,which will be described below. The selection unit 53 selects, among theimage capturing devices 10A-1, 10A-2, 10A-3, and 10A-4 arranged in thecircle having the radius r, the image capturing device 10A-1, which isarranged at the position closest to the point of interest P.

Then, the selection unit 53 extends a virtual line segment connectingthe point of interest P (predetermined position) and the image capturingdevice 10A-1 to divide the two-dimensional arrangement image generatedby the generation unit 56 into two areas. The two-dimensionalarrangement image obtained in the way described above is an example inthe case of the number of divisions being set to two.

Process for Selecting Image Capturing Device when Number of Divisions isTwo

FIG. 22 is a view schematically describing an example selection processbased on an angle between image capturing devices 10A when the number ofdivisions is two. As illustrated in the example in FIG. 22, the angle(azimuth angle) between the image capturing devices 10A-1 and 10A-2 is101 degrees. The angle between the image capturing devices 10A-1 and10A-3 is 121 degrees. The angle between the image capturing devices10A-1 and 10A-4 is 163 degrees. The distribution site A (area A) isdivided into two areas by a thick broken line obtained by extending thevirtual line segment connecting the point of interest P and the imagecapturing device 10A-1.

Relationship Between Point of Interest P and Image Capturing Deviceswhen Number of Divisions is Three

FIG. 23 is a view schematically describing an example relationshipbetween the point of interest P and the image capturing devices 10 whenthe number of divisions is three. For simplicity of description, thearrangement of the image capturing devices 10A at the distribution siteA illustrated in FIG. 23 is similar to that in FIG. 21. In this state,in response to the user who uses the communication terminal 70 operatingthe “point of interest” button 605 and the “number of divisions” button670, the selection unit 53 extends a virtual line segment connecting thepoint of interest P (predetermined position) and the image capturingdevice 10A-1 to divide the two-dimensional arrangement image generatedby the generation unit 56 into three areas such that the two-dimensionalarrangement image is separated by 120 degrees. The two-dimensionalarrangement image obtained in the way described above is an example inthe case of the number of divisions being set to three.

Process for Selecting Image Capturing Device when Number of Divisions isThree

FIG. 24 is a view schematically describing an example selection processbased on an angle between image capturing devices 10A when the number ofdivisions is three. The arrangement of the image capturing devices 10Aillustrated in FIG. 24 is similar to that of the image capturing devices10A illustrated in FIG. 22. In FIG. 24, the distribution site A (area A)is divided into three areas by thick broken lines based on the virtualline segment (thick broken line) connecting the point of interest P andthe image capturing device 10A-1 such that the thick broken lines eachhave an angle of 120 degrees.

Process for Selecting Image Capturing Device when Number of Divisions isThree (in Presence of Obstacle)

FIG. 25 is a view schematically describing an example selection processbased on an angle between image capturing devices 10A when the number ofdivisions is three. In the example illustrated in FIG. 25, a wall ispresent between the image capturing device 10A-3 and the point ofinterest P.

Relationship Between Point of Interest P and Image Capturing Deviceswhen Number of Divisions is Four

FIG. 26 is a view schematically describing an example relationshipbetween the point of interest P and the image capturing devices 10 whenthe number of divisions is four. The arrangement of the image capturingdevices 10A at the distribution site A illustrated in FIG. 26 is alsosimilar to that in FIGS. 21 and 23. In this state, in response to theuser who uses the communication terminal 70 operating the “point ofinterest” button 605 and the “number of divisions” button 670, theselection unit 53 extends a virtual line segment connecting the point ofinterest P (predetermined position) and the image capturing device 10A-1to divide the two-dimensional arrangement image generated by thegeneration unit 56 into four areas such that the two-dimensionalarrangement image is separated by 90 degrees. The two-dimensionalarrangement image obtained in the way described above is an example inthe case of the number of divisions being set to four.

FIG. 27 is a view schematically describing an example relationshipbetween the point of interest P and image capturing devices 10 arrangedin irregular positions. As illustrated in FIG. 27, the image capturingdevices 10 may be arranged in irregular positions at the distributionsite A. Also in this case, the selection unit 53 can divide, inaccordance with the number of divisions given by the user, that is, two,three, four, or the like, the area into the given number of divisions,on the basis of the positional relationship between the image capturingdevices 10 and the point of interest P in an area having the radius r.

Process or Operation According to Embodiment Session ParticipationProcess

Next, a process or operation of the image communication system 1according to an embodiment will be described with reference to FIGS. 28to 36. In the following description, a captured image of thedistribution site A is distributed to the communication terminal 70, byway of example. However, similar processing is performed fordistribution of an image from any other distribution site such as thedistribution site B. First, a process for participating in a specificcommunication session will be described with reference to FIGS. 28 and29. FIG. 28 is a sequence diagram illustrating an example process forparticipating in a specific communication session in the imagecommunication system 1. FIG. 29 is a view illustrating an example screenfor selecting a communication session.

First, a user (e.g., the user C1) at the viewing site C performs anoperation of displaying a selection screen for selecting a communicationsession. In response to the acceptance unit 72 accepting the operationof displaying the selection screen, the display control unit 74 of thecommunication terminal 70C causes the display 706 to display a selectionscreen 800 illustrated in FIG. 29 (step S11). The selection screen 800illustrated in FIG. 29 displays selection buttons 810 a, 810 b, 810 c,etc. indicating floors A1, B1, B2, etc. to be selected, respectively.The selection button 810 a and the other selection buttons areassociated with respective session IDs.

When the user C1 selects a selection button for the desired virtualfloor that is a distribution site (here, the selection button 810 a),the acceptance unit 72 accepts selection of a communication session(step S12). Then, the transmitting/receiving unit (acquisition unit) 71transmits to the communication management apparatus 50 a participationrequest to participate in the communication session with thedistribution site (step S13). Thus, the transmitting/receiving unit 51of the communication management apparatus 50 receives the participationrequest. The participation request includes a session ID indicating thecommunication session for which selection is accepted in step S12, andthe IP address of the communication terminal 70C, which is the requestsender terminal.

Then, the storing and reading unit 59 of the communication managementapparatus 50 adds, in the session management table (the sessionmanagement DB 5001, see FIG. 16A), the IP address received in step S13to the “IP address of participant terminal” field for the record of thesame session ID as the session ID received in step S13 to perform aprocess for participating in the communication session (step S14).Further, the storing and reading unit 59 reads, from the sessionmanagement DB 5001, the site ID associated with the session ID of thecommunication session for which the participation process is performed.Then, the storing and reading unit 59 searches the distribution-sitemanagement table (the distribution site management DB 5005, see FIG. 18)using the site ID read in step S14 as a search key to read thedistribution site information associated with the same site ID as theread site ID (step S15). Then, the transmitting/receiving unit 51transmits a participation request response to the communication terminal70C (step S16). Thus, the transmitting/receiving unit (acquisition unit)71 of the communication terminal 70C receives the participation requestresponse. The participation request response includes the session IDreceived in step S13, the distribution site information read in stepS15, and a participation process result. The participation processresult includes the site ID read in step S14. In the following, adescription will be given of a case in which the participation processis successful. Through a process similar to the process illustrated inFIG. 28, the communication terminal 70D at the viewing site D performs aprocess for participating in a communication session.

Process for Managing Image Type Information

Next, a process for managing the image type information will bedescribed with reference to FIG. 30. FIG. 30 is a sequence diagramillustrating a process for managing image type information in the imagecommunication system 1.

First, when a user at the distribution site A connects the imagecapturing device 10A to the distribution terminal 30A, the storing andreading unit 19 of the image capturing device 10A reads the GUID of theimage capturing device 10A from the storage unit 1000. Then, thecommunication unit 11 of the image capturing device 10A transmits theGUID of the image capturing device 10A to the distribution terminal 30A(step S31). Thus, the communication unit 37 of the distribution terminal30A receives the GUID of the image capturing device 10A.

Then, the determination unit 35 of the distribution terminal 30Adetermines whether the same vendor ID and product ID as the vendor IDand product ID in the GUID received in step S31 are managed in the imagecapturing device management table (the image capturing device managementDB 3001, see FIG. 15A) to determine the image type (step S32).Specifically, if the same vendor ID and product ID are managed in theimage capturing device management DB 3001, the determination unit 35determines that the image capturing device 10A is configured to capturea special image (here, a spherical image). By contrast, if the samevendor ID and product ID are not managed in the image capturing devicemanagement table (the image capturing device management DB 3001, seeFIG. 15A), the determination unit 35 determines that the image capturingdevice 10A is configured to capture a general image.

Then, the storing and reading unit 39 stores, in the image typemanagement table (the image type management DB 3002, see FIG. 15B), theIP address of the image capturing device 10A and image type informationindicating the determination result obtained in step S32 in associationwith each other (step S33). In this state, no image data ID isassociated with the IP address of the image capturing device 10A and theimage type information. The image type information is, for example, asource name determined in accordance with a predetermined naming rule oran image type (general image or special image).

Then, the transmitting/receiving unit 31 transmits to the communicationmanagement apparatus 50 an additional request for the image typeinformation (step S34). The additional request for the image typeinformation includes the IP address of the image capturing device 10Aand the image type information, which are stored in step S33, and thesite ID of the distribution site A. Thus, the transmitting/receivingunit 51 of the communication management apparatus 50 receives theadditional request for the image type information.

Then, the storing and reading unit 59 of the communication managementapparatus 50 searches the session management table (the sessionmanagement DB 5001, see FIG. 16A) using the site ID received in step S34as a search key to read the corresponding session ID (step S35).

Then, the generation unit 56 generates a unique image data ID (stepS36). Then, the storing and reading unit 59 stores, in the image typemanagement table (the image type management DB 5002, see FIG. 16B), thesession ID read in step S35, the image data ID generated in step S36,and the IP address of the image capturing device 10A and the image typeinformation received in step S34 in association with each other as a newrecord (step S37). Then, the transmitting/receiving unit 51 transmitsthe image data ID generated in step S36 to the distribution terminal 30A(step S38). Thus, the transmitting/receiving unit 31 of the distributionterminal 30A receives the image data ID.

Then, the storing and reading unit 39 of the distribution terminal 30Astores, in the image type management table (the image type management DB3002, see FIG. 15B), the image data ID received in step S38 inassociation with the IP address of the image capturing device 10A andthe image type information stored in step S33 (step S39).

On the other hand, the transmitting/receiving unit 51 of thecommunication management apparatus 50 transmits an image typeinformation addition notification to the communication terminal 70C(step S40). Thus, the transmitting/receiving unit (acquisition unit) 71of the communication terminal 70C receives the image type informationaddition notification. The image type information addition notificationincludes the image data ID generated in step S36, and the IP address ofthe image capturing device 10A and the image type information stored instep S37.

Then, the storing and reading unit 79 of the communication terminal 70Cstores, in the image type management table (the image type management DB7001, see FIG. 20A), the image data ID, the IP address of the imagecapturing device 10A, and the image type information, which are receivedin step S40, in association with each other as a new record (step S41).Accordingly, the distribution terminal 30A and the communicationterminal 70C can share the same information in the image type managementDBs 3002 and 7001, respectively. The image type information additionnotification is also transmitted to another communication terminal,namely, the communication terminal 70D, and is stored in the image typemanagement DB 7001 of the communication terminal 70D.

Process for Communicating Captured Image Data

Next, a process for transmitting the captured image data and audio dataobtained at the distribution site A to the communication terminals 70(i.e., the communication terminals 70C and 70D) via the communicationmanagement apparatus 50 will be described with reference to FIG. 31.FIG. 31 is a sequence diagram illustrating an example process forcommunicating captured image data and audio data in the imagecommunication system 1. FIG. 31 illustrates an example in which capturedimage data acquired by one image capturing device 10 is distributed tothe communication terminals 70. However, similar processing is performedwhen a plurality of pieces of captured image data acquired by otherimage capturing devices 10 arranged in the distribution site aredistributed.

First, the communication unit 11 of the image capturing device 10Atransmits to the distribution terminal 30A captured image data acquiredby capturing an object or surroundings such as scenery and audio dataacquired by collecting sounds (step S51). Thus, the communication unit37 of the distribution terminal 30A receives the captured image data andthe audio data. In this case, since the image capturing device 10A iscapable of obtaining two hemispherical images from which a sphericalimage is generated, as illustrated in FIGS. 3A and 3B, the capturedimage data includes data of two hemispherical images.

Then, the transmitting/receiving unit 31 of the distribution terminal30A transmits to the communication management apparatus 50 the capturedimage data and the audio data sent from the image capturing device 10A(step S52). Thus, the transmitting/receiving unit 51 of thecommunication management apparatus 50 receives the captured image data,the audio data, and the image data ID. An image data ID for identifyingthe captured image data to be transmitted and received is alsotransmitted and received.

Then, the transmitting/receiving unit 51 of the communication managementapparatus 50 transmits the captured image data and the audio data to thecommunication terminals (the communication terminals 70C and 70D)participating in the same session as the session in which thedistribution terminal 30A is participating (steps S53 and S54). Thus,the transmitting/receiving unit (acquisition unit) 71 of each of thecommunication terminals 70C and 70D receives the captured image data andthe audio data. The image data ID for identifying the captured imagedata to be transmitted and received is also transmitted and received.

Process for Displaying Point of Interest P

Next, a process for displaying an image of the point of interest P in adistribution site, which is designated by a user at a viewing site, willbe described with reference to FIGS. 32 to 36. FIG. 32 is a sequencediagram illustrating an example process for displaying the point ofinterest P in the image communication system 1. FIG. 32 illustrates anexample in which the communication terminal 70C at the viewing site Cdisplays a captured image distributed from the distribution terminal30A. Similar processing is performed when the captured image isdisplayed on the communication terminal 70D at the viewing site D, whichis another viewing site.

First, the display control unit 74 of the communication terminal 70Cuses the distribution site information received in step S16 to cause thedisplay 706 to display a site display screen 600 indicating thesituation of the distribution site A (step S71). FIG. 34 is a viewillustrating an example site display screen displayed on thecommunication terminal 70C. The site display screen 600 illustrated inFIG. 34 displays a map image indicating the situation of a distributionsite.

The site display screen 600 illustrated in FIG. 34 includes a tour image610 for viewing a spherical image of the distribution site to enableremote viewers to see the distribution site, a point-of-view change icon630 for changing the point of view to a predetermined image capturingpoint in the tour image 610, a cross (or a star or any other suitablemark) 635 indicating the point of interest P, a schematic image 650indicating a schematic diagram of the distribution site, the “point ofinterest” button 605, a “cancel” button 609, and the “number ofdivisions” button 670. The “point of interest” button 605 is operated todisplay the point of interest P in the distribution site. The “cancel”button 609 is operated to terminate the viewing of the distributionsite. The “number of divisions” button 670 is used to divide an area ofthe distribution site centered on the point of interest P in thedistribution site (from the point of interest P). The “number ofdivisions” button 670 is operated to display a number-of-divisionssetting screen 675, which is a pop-up window in this example, near the“number of divisions” button 670. The number-of-divisions setting screen675 includes an input field that follows “Number of divisions=” to enterthe number of divisions. The user enters the desired value (e.g., 2, 3,or the like) in the input field and then operates an “OK” button. Theuser can operate a “cancel” button to change or remove the enteredvalue.

The tour image 610 and the schematic image 650 are displayed using themap image data included in the distribution site information received instep S16. The tour image 610 is an image of the distribution site, whichis captured by the image capturing device 10 in advance. The user C1 canoperate the point-of-view change icon 630 to understand the generalarrangement and the like of the distribution site. The tour image 610 isdescribed herein as being displayed using the map image data included inthe distribution site information received in step S16. However, thetour image 610 may be configured to display captured image data of thedistribution site, which is received in real time.

The cross 635 in the schematic image 650 functions as a point-of-viewposition icon indicating the current display position of the tour image610. For example, the user C1 operates the point-of-view change icon 630using a pointer p1 while viewing the tour image 610 and the schematicimage 650. As a result, the user C1 is able to view a desired portion inthe distribution site.

Then, the user C1 operates the point-of-view change icon 630 using thepointer p1 and operates the “point of interest” button 605 such that theacceptance unit 72 accepts the designation (input) of the point ofinterest P (step S72). The acceptance unit 72 may accept the designationof the point of interest P in response to a click or double-clickoperation of the user C1 using a mouse, which is an example of an inputdevice.

The user C1 further operates the pointing device 712 to operate the“number of divisions” button 670. As a result, the display control unit74 displays the number-of-divisions setting screen 675 on the sitedisplay screen 600. In response to the display of thenumber-of-divisions setting screen 675, the user C1 enters the number ofdivisions n (e.g., 2, 3, or the like) and then operates the “OK” buttonsuch that the acceptance unit 72 accepts the input of the number ofdivisions (step S73).

Then, the point-of-interest specifying unit 77 specifiespoint-of-interest coordinates, which are the coordinates of the point ofinterest P for which the designation is accepted in step S72 (step S74).The point-of-interest coordinates specified by the point-of-interestspecifying unit 77 are the coordinates of the central point of the tourimage 610 being displayed when the designation of the point of interestP is accepted in step S72. The point-of-interest coordinates may becoordinates indicating the position of the pointer p1 in the tour image610.

The transmitting/receiving unit (acquisition unit) 71 transmits to thecommunication management apparatus 50 an image-capturing-deviceselection request indicating a request for selecting an image capturingdevice 10 (step S75). Thus, the transmitting/receiving unit 51 of thecommunication management apparatus 50 receives theimage-capturing-device selection request transmitted from thecommunication terminal 70C. The image-capturing-device selection requestincludes the point-of-interest coordinates specified in step S74, thenumber of divisions n received in step S73, and the site ID received instep S16.

Process for Selecting Image Capturing Device

Then, the communication management apparatus 50 executes a process forselecting an image capturing device 10 in response to theimage-capturing-device selection request received in step S75 (stepS76). The process in step S76 will be described in detail with referenceto FIGS. 33 and 21 to 26. FIG. 33 is a flowchart illustrating an exampleprocess for selecting an image capturing device.

First, the storing and reading unit 59 searches the distribution-sitemanagement table (the distribution site management DB 5005, see FIG. 18)using the site ID received in step S75 as a search key to read thedistribution site information associated with the same site ID as thereceived site ID (step S101).

Then, the selection unit 53 divides the distribution site centered onthe point of interest P (predetermined position) into a plurality ofareas (from the point of interest P), based on the point-of-interestcoordinates and the number of divisions n both received in step S75(step S102). Specifically, the selection unit 53 divides the entire areaof the distribution site into two areas using the receivedpoint-of-interest coordinates as a starting point, based on the sitecoordinate information included in the distribution site informationread in step S101 and the number-of-division information (the number ofdivisions n) accepted in step S73. In the example illustrated in FIG.21, the point of interest P designated by the user C1 is indicated by ablack circle.

Then, the storing and reading unit 59 searches the arrangementinformation management table (the arrangement information management DB5004, see FIG. 17B) using the site ID received in step S75 as a searchkey to read the arrangement information associated with the same site IDas the received site ID (step S103).

Then, the selection unit 53 selects image capturing devices 10 arrangednear the point-of-interest coordinates indicating the coordinates of thepoint of interest P, based on the coordinate values indicated by thearrangement information read in step S103 (step S104). Specifically, theselection unit 53 extends a virtual line segment connecting the point ofinterest P (predetermined position) and the image capturing device 10A-1to divide the two-dimensional arrangement image generated by thegeneration unit 56 into two areas. To draw a circle having the radius r,the selection unit 53 sets the value of the radius r such that a circlehaving the radius r includes at least two image capturing devices 10that are relatively close to the point of interest P. To set the valueof the radius r, the selection unit 53 can calculate, from thecoordinate values of the point of interest P and the respectivecoordinate values of the image capturing devices 10 that are arrangednear the point of interest P, the distance between each of the imagecapturing devices 10 and the point of interest P. Then, the selectionunit 53 selects the image capturing devices 10A-1, 10A-2, 10A-3, and10A-4 as in-range image capturing devices 10 arranged near thepoint-of-interest coordinates indicating the coordinates of the point ofinterest P.

In this embodiment, the image capturing device 10A-1, which is arrangedat the position closest to the point of interest P and selected in theprocessing of step S104, is referred to as “reference image capturingdevice”. The reference image capturing device 10A-1 functions as, or hasmeans for functioning as, an example of a first image capturing device.

Then, the selection unit 53 executes a process for selecting asupplementary image capturing device from among the in-range imagecapturing devices 10 (step S105). The supplementary image capturingdevice is capable of capturing a portion of an object at the point ofinterest P for which an image is difficult for the reference imagecapturing device 10A-1 to satisfactorily capture. The supplementaryimage capturing device functions as an example of a second imagecapturing device.

Process for Selecting Supplementary Image Capturing Device

FIG. 35 is a flowchart illustrating an example process for selecting asupplementary image capturing device. First, the selection unit 53 drawsa predetermined range (e.g., a circle having a radius r) centered on thepoint of interest P. Then, the selection unit 53 selects, among theimage capturing devices 10A-1, 10A-2, 10A-3, and 10A-4 arranged in thecircle having the radius r, the image capturing device 10A-1, which isarranged at the position closest to the point of interest P (stepS105-1). At this time, in the example illustrated in FIG. 21, theselection unit 53 calculates the distance between the coordinate valuesof each of the image capturing devices 10A-1, 10A-2, 10A-3, and 10A-4and the point-of-interest coordinates. Then, the selection unit 53determines that the image capturing device 10A-1, which is arranged atthe position closest to the coordinates of the point of interest P (orhas the shortest distance from the point of interest P), is thereference image capturing device 10A-1 that is closest to the point ofinterest P in the predetermined range, and holds the identificationinformation of the image capturing device 10A-1 (e.g., the IP address ofthe image capturing device 10A-1) and the calculated distance in thestorage unit 5000. The distance between the coordinate values of each ofthe plurality of image capturing devices 10A and the point-of-interestcoordinates is calculated using, for example, any desired known methodfor determining the distance in a straight line connecting thecoordinates of each of the plurality of image capturing devices 10A andthe coordinates of the point of interest P.

Then, the selection unit 53 calculates the angle between the referenceimage capturing device 10A-1 and another image capturing device near thepoint of interest P (step S105-2). Specifically, the selection unit 53searches the arrangement information management table (the arrangementinformation management DB 5004, see FIG. 17B) using the IP address ofeach image capturing device as a search key to read the coordinatevalues of the corresponding image capturing device. Then, the selectionunit 53 calculates angles between lines connecting the respective imagecapturing devices and the point of interest P. As a result, in theexample illustrated in FIG. 22 described above, the angle (azimuthangle) at the point of interest P between the image capturing devices10A-1 and 10A-2 is 101 degrees. The angle at the point of interest Pbetween the image capturing devices 10A-1 and 10A-3 is 121 degrees. Theangle at the point of interest P between the image capturing devices10A-1 and 10A-4 is 163 degrees. The selection unit 53 holds thecalculated angles in the storage unit 5000 in a manner similar to thatfor the distances calculated in the processing of step S105-2. Theselection unit 53 can determine these angles by using, for example, theinverse trigonometric functions for the respective coordinate values.The distribution site A (area A) is divided into two areas by a thickbroken line obtained by extending a virtual line segment connecting thepoint of interest P and the image capturing device 10A-1.

After calculating the angles between the image capturing devices in stepS105-2, the selection unit 53 selects the following image capturingdevices as candidate supplementary image capturing devices (second imagecapturing devices) that perform a supplementary operation to capture animage of a portion that is difficult for the reference image capturingdevice to capture (step S105-3). Specifically, in a case in which thenumber of divisions is n, where n is a natural number greater than orequal to 2, the selection unit 53 selects, as a candidate supplementaryimage capturing device (second image capturing device), an imagecapturing device arranged at the angle closest to an angle obtained bydividing 360 degrees by the number of divisions n from a virtual linesegment connecting the reference image capturing device (first imagecapturing device) and the point of interest P.

Case in which Number of Divisions is Two

Specifically, as illustrated in FIG. 22, the selection unit 53 selectsan image capturing device arranged such that the angle between the imagecapturing device and the reference image capturing device 10A-1 isclosest to an angle obtained by 360°/n, that is, 360°/2=180°, as acandidate supplementary image capturing device to the reference imagecapturing device 10A-1. That is, the selection unit 53 selects an imagecapturing device arranged such that the angle between the imagecapturing device and the reference image capturing device 10A-1 isclosest to an angle of 180° as a candidate supplementary image capturingdevice. In this case, images captured by the reference image capturingdevice 10A-1 and the supplementary image capturing device 10A-4 cancover, for example, the most sides (different aspects) of the point ofinterest P. Thus, the selection unit 53 selects the image capturingdevice 10A-4 as a candidate supplementary image capturing device to thereference image capturing device 10A-1. However, the selection unit 53may use an angle of 180 degrees with some margin (θ) as a determinationcriterion for selecting a supplementary image capturing device. That is,the selection unit 53 may select an image capturing device arranged suchthat the angle between the image capturing device and the referenceimage capturing device 10A-1 is closest to an angle of 180°±θ as acandidate supplementary image capturing device.

In the following, a description will be given of a specific example ofcoefficient computation using a data table and selection of asupplementary image capturing device. As in the weighting coefficientmanagement table (the weighting coefficient management DB 5006, see FIG.19A) and the specific weighting coefficient management table (thespecific weighting coefficient management DB 5007, see FIG. 19B)described below, the image capturing devices 10A-1, 10A-2, 10A-3, and10A-4 are represented by CAM1, CAM2, CAM3, and CAM4, respectively, forconvenience of description.

In a case in which four image capturing devices, CAM1, CAM2, CAM3, andCAM4, are arranged in a circle having a radius r centered on the pointof interest P illustrated in FIG. 22, the selection unit 53 selects thefollowing combination of two image capturing devices among the fourimage capturing devices. At this time, the angles at the point ofinterest P between the image capturing devices are as follows.

1. Case in which the selection unit 53 selects CAM1 and CAM2:

Angle between CAM1 and CAM2=101 degrees

2. Case in which the selection unit 53 selects CAM1 and CAM3:

Angle between CAM1 and CAM3=121 degrees

3. Case in which the selection unit 53 selects CAM1 and CAM4:

Angle between CAM1 and CAM4=163 degrees

The optimum positions of two image capturing devices that can view(capture) the point of interest P from different aspects are obtainedwhen the angle at the point of interest P between the two imagecapturing devices is 180 degrees. Accordingly, the selection unit 53calculates the absolute value of the difference between 180 degrees andthe angle at the point of interest P between each pair of imagecapturing devices and selects the combination of image capturing devicesfor which the absolute value is smallest. The angles between the pairsof image capturing devices illustrated in FIG. 22 are as follows:

∥01−180|=79 in the case 1;

|121−180|=59 in the case 2; and

|163−180|=17 in the case 3.

The combination of two image capturing devices between which theabsolute value of the difference is smallest is obtained as imagecapturing devices arranged such that the angle at the point of interestP therebetween is closest to 180°±θ. Accordingly, the selection unit 53selects CAM4 in the case 3 as a candidate supplementary image capturingdevice to CAM1, which is the reference image capturing device.

In this state, the selection unit 53 performs coefficient computationusing at least one data table among the weighting coefficient managementtable (the weighting coefficient management DB 5006, see FIG. 19A) andthe specific weighting coefficient management table (the specificweighting coefficient management DB 5007, see FIG. 19B) (step S105-4).That is, the selection unit 53 determines the image capturing device10A-4 (CAM4) as a candidate supplementary image capturing device andthereafter performs the following three computations including, forexample, the reference image capturing device 10A-1 (CAM1).

1. |101−180|*1.0=79

2. |121−180|*1.0=59 (In this case, however, since an obstacle such as awall is absent in front of CAM3, computation is performed with thecoefficient in the data table being set to 1.0)

3. |163−180|*1.0=17

Accordingly, also in the processing of step S105-4, the selection unit53 determines the combination for which the computation result in thecase 3 in which the angle between CAM1 and CAM4 is 1630 has the smallestvalue, and determines that a supplementary image capturing device to theimage capturing device 10A-1 (CAM1) serving as the reference imagecapturing device is the image capturing device 10A-4 (CAM4). Then, theselection unit 53 exits the flowchart.

Case in which Number of Divisions is Three

Next, as illustrated in FIG. 23, a case in which the number of divisionsis three will be described. The selection unit 53 selects an imagecapturing device arranged such that the angle between the imagecapturing device and the reference image capturing device 10A-1 isclosest to an angle obtained by 360°/n, that is, 360°/3=120°, as acandidate supplementary image capturing device to the reference imagecapturing device 10A-1 (step S105-3). However, the selection unit 53 mayuse an angle of 120 degrees with some margin (θ) as a determinationcriterion for selecting a supplementary image capturing device. That is,the selection unit 53 may select an image capturing device arranged suchthat the angle between the image capturing device and the referenceimage capturing device 10A-1 is closest to an angle of 120°±θ as acandidate supplementary image capturing device. As described above, in acase in which the number of divisions is three, the selection unit 53selects two image capturing devices as candidate supplementary imagecapturing devices to the reference image capturing device 10A-1.

In the following, a description will be given of a specific example ofcoefficient computation using a data table and selection of asupplementary image capturing device. As in the case in which the numberof divisions is two, the image capturing devices 10A-1, 10A-2, 10A-3,and 10A-4 are represented by CAM1, CAM2, CAM3, and CAM4, respectively,for convenience of description.

In a case in which four image capturing devices, CAM1, CAM2, CAM3, andCAM4, are arranged in a circle having a radius r centered on the pointof interest P illustrated in FIG. 23, the selection unit 53 selects thefollowing combination of three image capturing devices among the fourimage capturing devices. At this time, the angles at the point ofinterest P between the image capturing devices are as follows.

1. Case in which the selection unit 53 selects CAM1, CAM2, and CAM3:

Angle between CAM1 and CAM2=101 degrees;

Angle between CAM2 and CAM3=138 degrees; and

Angle between CAM3 and CAM1=121 degrees.

2. Case in which the selection unit 53 selects CAM1, CAM2, and CAM4:

Angle between CAM1 and CAM2=101 degrees;

Angle between CAM2 and CAM4=71 degrees; and

Angle between CAM4 and CAM1=188 degrees.

3. Case in which the selection unit 53 selects CAM1, CAM3, and CAM4:

Angle between CAM1 and CAM3=121 degrees;

Angle between CAM3 and CAM4=67 degrees; and

Angle between CAM4 and CAM1=172 degrees.

The optimum positions of three image capturing devices that can view(capture) the point of interest P from different aspects are obtainedwhen the angle at the point of interest P between the three imagecapturing devices is 120 degrees. Accordingly, the selection unit 53calculates the absolute value of the difference between 120 degrees andthe angle at the point of interest P between the three image capturingdevices and selects a combination of image capturing devices for whichthe absolute value is smallest. The angles between the image capturingdevices illustrated in FIG. 24 are as follows:

|101−120|+|138−120|+|121−120|=38 in the case 1;

|101−120|+|71−120|+|188−120|=136 in the case 2; and

|121−120|+|67−120|+|72−120|=106 in the case 3.

The combination of three image capturing devices between which theabsolute value of the difference is smallest is obtained as imagecapturing devices arranged such that the angle at the point of interestP therebetween is closest to 120°±θ. Accordingly, the selection unit 53selects CAM2 and CAM3 in the case 1 as candidate supplementary imagecapturing devices to CAM1, which is the reference image capturingdevice.

In this state, as in the case in which the number of divisions is two,the selection unit 53 performs coefficient computation using at leastone data table among the weighting coefficient management table (theweighting coefficient management DB 5006, see FIG. 19A) and the specificweighting coefficient management table (the specific weightingcoefficient management DB 5007, see FIG. 19B) (step S105-4). That is,the selection unit 53 determines the image capturing device 10A-2 (CAM2)and the image capturing device 10A-3 (CAM3) as candidate supplementaryimage capturing devices and thereafter performs the followingcomputation.

Case in which Obstacle is Present in Front of Image Capturing Device

In a case in which an obstacle or the like is present between the pointof interest P and an image capturing device when a supplementary imagecapturing device is to be selected, the selection unit 53 performscalculation using a predetermined coefficient so that a combinationincluding the image capturing device is not likely to be selected. Asillustrated in FIG. 25, it is assumed that a wall is present between theimage capturing device 10A-3 and the point of interest P and that it isdifficult for the image capturing device 10A-3 to capture an image ofthe point of interest P.

In this case, the weighting coefficient management table (the weightingcoefficient management DB 5006, see FIG. 19A) is used for each term ofthe calculation equation described above. A coefficient in the weightingcoefficient management table can be set to a desired value in advance bythe administrator who manages the communication management apparatus 50or the like. The administrator or the like checks the plurality of imagecapturing devices arranged at the distribution site. Upon determiningthat a wall, an obstacle, or the like is present near an image capturingdevice, the administrator or the like sets a coefficient for thecorresponding image capturing device to a larger value than coefficientsfor the other image capturing devices. As a result, the selection unit53 performs a process for removing an image capturing device hidden bythe wall, the obstacle, or the like among the in-range image capturingdevices from candidate supplementary image capturing devices. Thisfurther enables the user who uses the communication terminal 70 toeasily observe an object present at the point of interest P. Inaddition, upon detection of fault such as malfunctioning of an imagecapturing device among the in-range image capturing devices, theselection unit 53 can remove the image capturing device from candidatesupplementary image capturing devices. The reason for this is that theselection unit 53 selects a supplementary image capturing device havinga small calculation result based on the computation equation describedabove.

The calculation results of the image capturing devices illustrated inFIG. 25 are as follows.

1.  101 − 120 * 1.0 + 138 − 120 * 10.0 + 121 − 120 * 10.0 = 2.092.  101 − 120 * 1.0 + 71 − 120 * 1.0 * 188 − 120 * 1.0 = 1363.  121 − 120 * 10.0 + 67 − 120 * 1.0 + 172 − 120 * 1.0 = 115 

As a result of the calculation, the selection unit 53 selects thecombination of CAM1, CAM3, and CAM4 in the case 3 having the smallestvalue of the calculation results. However, a wall is present in front ofCAM3, which is not suitable for practical use. Thus, the selection unit53 cooperates with the determination unit 55 and performs the followingdetermination and selection. In this case, the selection unit 53 selectsthe combination having the second smallest value of the calculationresults. That is, the selection unit 53 selects CAM1 (the imagecapturing device 10A-1) as the reference image capturing device and CAM2(the image capturing device 10A-2) and CAM4 (the image capturing device10A-4) as candidate supplementary image capturing devices. As a result,CAM3 (the image capturing device 10A-3) hidden by the obstacle or thelike is not selected, and an object present at the point of interest Pis easily observed. However, as in the example illustrated in FIG. 25,even when CAM4 (the image capturing device 10A-4) is selected as acandidate supplementary image capturing device, it may still bedifficult to capture images that cover the most sides (differentaspects) of the point of interest P. In this case, the selection unit 53may stop the selection of a supplementary image capturing device orprovide a message or the like that prompts the user who uses thecommunication terminal 70 to stop the selection process or designate adifferent point of interest.

Case in which Number of Divisions is Four

An example in a case in which the number of divisions n is four isillustrated in FIG. 26. In the illustrated example, the selection unit53 selects the image capturing device 10 arranged at the positionclosest to a central angle of 90°±θ centered on the point of interest Pin a circle having a radius r centered on the point of interest P as asupplementary image capturing device that is supplementary to thereference image capturing device 10A-1. In the example illustrated inFIG. 26, three supplementary image capturing devices, namely, the imagecapturing devices 10A-2, 10A-3, and 10A-4, are selected.

Case in which Number of Divisions is Five or More

Also in a case in which the number of divisions is five or more, theselection unit 53 selects, as a candidate supplementary image capturingdevice (second image capturing device), an image capturing devicearranged at the angle closest to an angle obtained by dividing 360degrees by the number of divisions n from a virtual line segmentconnecting the reference image capturing device (first image capturingdevice) and the point of interest P (step S105-3). Thereafter, theselection unit 53 can perform coefficient computation using at least onedata table among the weighting coefficient management table (theweighting coefficient management DB 5006, see FIG. 19A) and the specificweighting coefficient management table (the specific weightingcoefficient management DB 5007, see FIG. 19B) (step S105-4), and selecta reference image capturing device and a supplementary image capturingdevice that is supplementary to the reference image capturing device.

The processing of steps S105-3 and S105-4 described above may beperformed in reverse order by the communication management apparatus 50.That is, after coefficient computation is executed using the data tabledescribed in step S105-4, a process may be executed in step S105-3 forselecting an image capturing device (supplementary image capturingdevice) arranged such that the angle between the image capturing device(supplementary image capturing device) and the reference image capturingdevice among the selected image capturing devices is closest to 360°/n.However, if an obstacle such as a wall is present near the selectedsupplementary image capturing device or if the selected supplementaryimage capturing device is malfunctioning, the selection unit 53 and thedetermination unit 55 may remove such an image capturing device fromcandidate supplementary image capturing devices and determine whetheranother image capturing device is available as an alternative candidatesupplementary image capturing device.

Instead of the weighting coefficient management table (the weightingcoefficient management DB 5006, see FIG. 19A), the communicationmanagement apparatus 50 may manage the specific weighting coefficientmanagement table (the specific weighting coefficient management DB 5007,see FIG. 19B) in which a combination of specific image capturing devicesis removed, and perform calculation with a coefficient set to 1.0 for acombination of a reference image capturing device and a supplementaryimage capturing device that is not managed in the specific weightingcoefficient management table. This method only requires theadministrator who manages the communication management apparatus 50 toset and manage a specific weighting coefficient for an image capturingdevice for which setting is to be performed. As a result, theadministrator can save their work at the distribution site or the likewhere a large number of image capturing devices are arranged forprevious setting of the weighting coefficient in these image capturingdevices.

The process for selecting a reference image capturing device and asupplementary image capturing device is also applicable to a condition,as illustrated in FIG. 27, in which the image capturing devices 10 arearranged at irregular intervals in the distribution site.

Referring back to FIG. 33, the generation unit 56 generates, for each ofthe image capturing devices 10A selected in step S104,predetermined-area information corresponding to the point-of-interestcoordinates received in step S75 (step S106). Specifically, thegeneration unit 56 generates predetermined-area information fordisplaying a predetermined-area image (an image in perspectiveprojection) such that the point of interest indicated by thepoint-of-interest coordinates is located at the center of a sphericalimage captured by each of the selected image capturing devices 10A. Itis assumed here that the radial distance (r) and the polar angle (θ)have predetermined values set in advance. The use of the predeterminedvalues indicates that the image in perspective projection is processedwith the angle of view and the elevation angle thereof being constant.The azimuth angle (ϕ) can be calculated as a relative value between thearrangement direction of the image capturing device 10A indicated in thearrangement information and the direction from the image capturingdevice 10A to the point of interest indicated by the point-of-interestcoordinates. Then, the generation unit 56 generates, based on thecalculated results, predetermined-area information for displaying apredetermined-area image (displaying an image in perspective projection)centered on the point-of-interest coordinates from a spherical imageacquired by the image capturing device 10A. The generation unit 56executes the process described above on each of the image capturingdevices 10A selected in step S104. The arrangement position (coordinatevalues) and the arrangement direction of the image capturing device 10A(the direction in which the front surface of the image capturing device10A faces) are set in advance in the arrangement information managementDB 5004 by the administrator or the like. In the processing of step S106described above, to obtain a combination of the IP address of an imagecapturing device and predetermined-area information, the storing andreading unit 59 and the selection unit 53 use the predetermined-areamanagement table (the predetermined-area management DB 5003, see FIG.17A) and the arrangement information management table (the arrangementinformation management DB 5004, see FIG. 17B), which are managed by thecommunication management apparatus 50, to generate correspondingpredetermined-area information.

Then, the storing and reading unit 59 stores the predetermined-areainformation generated in step S106 in the predetermined-area managementtable (the predetermined-area management DB 5003, see FIG. 17A) inassociation with the IP address of the corresponding image capturingdevice 10A (step S107).

As described above, the communication management apparatus 50 can selectan image capturing device 10 that captures an image of the point ofinterest P, based on the point-of-interest coordinates transmitted fromthe communication terminal 70 and the arrangement information of theplurality of image capturing devices 10, and calculate the angle of viewof the selected image capturing device 10A to display the point ofinterest P.

In this embodiment, the generation unit 56 functions as, or has meansfor functioning as, an example of generation means. Further, thegeneration unit 56 that generates predetermined-area information may beincluded in, instead of the communication management apparatus 50,another apparatus disposed between the communication managementapparatus 50 and the communication terminal 70.

Referring back to FIG. 32, the transmitting/receiving unit 51 of thecommunication management apparatus 50 transmits to the communicationterminal 70C a selection result notification that is a result of theprocess in step S76 (step S77). Thus, the transmitting/receiving unit(acquisition unit) 71 of the communication terminal 70C receives theselection result notification transmitted from the communicationmanagement apparatus 50. The selection result notification includes setsof pieces of predetermined-area information generated in step S106 andIP addresses of image capturing devices 10, the number of whichcorresponds to the number of pieces of predetermined-area informationgenerated in step S106. As described above, the transmitting/receivingunit (acquisition unit) 71 of the communication terminal 70C acquirespredetermined-area information indicating a predetermined area includingthe point of interest P for which the designation is accepted in stepS72.

As described above, the transmitting/receiving unit 51 transmits to thecommunication terminal 70 predetermined-area information (firstpredetermined-area information) indicating a predetermined areaincluding a predetermined position in an image captured by the imagecapturing device 10A-1, which is the reference image capturing device10A-1 (first image capturing device) arranged at the shortest distancefrom the point of interest P (predetermined position) among a pluralityof image capturing devices arranged in an area of a distribution site.In addition, the transmitting/receiving unit 51 transmits to thecommunication terminal 70 predetermined-area information (secondpredetermined-area information) indicating a predetermined areaincluding the predetermined position in an image captured by asupplementary image capturing device, which is a supplementary imagecapturing device (second image capturing device) arranged at the angleclosest to an angle obtained by dividing the area of the distributionsite in accordance with the number of divisions indicated innumber-of-division information. In a case in which the number ofdivisions is two, in the example illustrated in FIG. 21, the imagecapturing device 10A-4 is a supplementary image capturing device (secondimage capturing device) arranged at the angle closest to the angleobtained by dividing the area of the distribution site in accordancewith the number of divisions. In a case in which the number of divisionsis three, in the example illustrated in FIG. 23, the image capturingdevices 10A-2 and 10A-3 are supplementary image capturing devices(second image capturing devices) arranged at the angle closest to theangle obtained by dividing the area of the distribution site inaccordance with the number of divisions.

The transmitting/receiving unit 51 of the communication managementapparatus 50 can transmit the first predetermined-area information andthe second predetermined-area information to the communication terminal70 at the same time or different times. When the transmitting/receivingunit 51 transmits the first predetermined-area information and thesecond predetermined-area information to the communication terminal 70at different times, the first predetermined-area information and thesecond predetermined-area information may be transmitted in any order.

Then, the storing and reading unit 79 of the communication terminal 70Cstores the predetermined-area information received in step S77 in thepredetermined-area management table (the predetermined-area managementDB 7002, see FIG. 20B) in association with the IP addresses of the imagecapturing devices 10 (step S78).

Then, the transmitting/receiving unit (acquisition unit) 71 transmits acaptured-image request to the communication management apparatus 50 toacquire an image captured by the selected image capturing device 10(step S79). Thus, the transmitting/receiving unit 51 of thecommunication management apparatus 50 receives the captured-imagerequest transmitted from the communication terminal 70. Thecaptured-image request includes the image data ID acquired in step S40.

Upon receipt of the captured-image request, the storing and reading unit59 of the communication management apparatus 50 searches the image typemanagement table (the image type management DB 5002, see FIG. 16B) usingthe image data ID as a search key to read the corresponding source name(image type information) (step S80). As a result, image datarepresenting the captured image corresponding to the image data ID isspecified.

The transmitting/receiving unit 51 transmits to the communicationterminal 70C the image data representing the captured image that is read(step S81). Thus, the transmitting/receiving unit 71 of thecommunication terminal 70C receives the image data transmitted from thecommunication management apparatus 50. The image data includes the imagedata ID of the captured image and audio data.

Then, in response to a predetermined input operation of the user C1, theacceptance unit 72 accepts selection of a display image to be displayedon the communication terminal 70C within the captured image datareceived in step S53 (step S82). In a case in which the communicationterminal 70C is capable of simultaneously displaying a plurality ofcaptured images or in a case in which fewer captured images than thenumber of images simultaneously displayable on the communicationterminal 70C are received, the processing of step S78 may be omitted.

Then, to display an image of a predetermined area specified in thepredetermined-area information corresponding to the display image forwhich selection is accepted in step S82, the image and audio processingunit 73 applies perspective projection conversion using thepredetermined-area information received in step S77 to generate apredetermined-area image (step S83). As a result, the communicationterminal 70C can generate a predetermined-area image including the pointof interest P designated by the user in a spherical image that is animage captured by the image capturing device 10 selected by thecommunication management apparatus 50.

Then, the display control unit 74 causes the display 706 to display thepredetermined-area image generated in step S83 (step S84). FIG. 36illustrates an example of the display screen displayed on thecommunication terminal 70C in step S84. A display screen 900 illustratedin FIG. 36 displays a predetermined-area image X generated in step S83.The user C1 checks the predetermined-area image X corresponding to theposition of the point of interest P in the distribution site, which isdesignated using the site display screen 600. As a result, the user C1is able to check the details of the current situation at the position ofthe point of interest P in the distribution site.

As described above, in the image communication system 1, an imagecapturing device 10 suitable to view the point of interest P designatedby a user at a viewing site can be selected using the communicationmanagement apparatus 50. In the image communication system 1,furthermore, predetermined-area information capable of displaying animage including the point of interest P, which is captured by theselected image capturing device 10, is transmitted from thecommunication management apparatus 50 to the communication terminal 70.As a result, the image indicating the point of interest P can bedisplayed on the communication terminal 70.

In this embodiment, the selection unit 53 divides the distribution siteinto two areas or three areas, by way of example. However, the selectionunit 53 may divide the distribution site into any number of areas inaccordance with the number of image capturing devices 10 or distributionterminals 30 arranged in the distribution site, the network bandwidth ofthe communication network 100, the number of images simultaneouslyreceivable and displayable at the communication terminal 70, or thelike.

In the image communication system 1 according to this embodiment, whenprocessing such as the processing of steps S75 to S77 or the processingof steps S79 to S81 described above is executed, another apparatus orthe like may be present between the communication management apparatus50 (communication management apparatus) and the communication terminal70 (communication terminal). That is, information may be transmitted andreceived between the communication management apparatus 50 and thecommunication terminal 70 via another apparatus.

Modification of Process for Displaying Point of Interest P

Next, a modification of the process for displaying the point of interestP in the image communication system 1 will be described with referenceto FIG. 37. FIG. 37 is a sequence diagram illustrating a modification ofthe process for displaying the point of interest P in the imagecommunication system 1. FIG. 37 illustrates an example in which theprocess for selecting an image capturing device 10 described above isexecuted by the communication terminal 70C at the viewing site C. Sincethe processing of steps S201 to S204 is similar to the processing ofsteps S71 to S74 in FIG. 32, the description thereof will be omitted.

In FIG. 37, the transmitting/receiving unit (acquisition unit) 71transmits to the communication management apparatus 50 anarrangement-information acquisition request, which is a request foracquiring arrangement information indicating the arrangement positionsof the image capturing devices 10 (step S205). Thus, thetransmitting/receiving unit 51 of the communication management apparatus50 receives the arrangement-information acquisition request transmittedfrom the communication terminal 70C. The arrangement-informationacquisition request includes the site ID received in steps S14 and S16.

Then, the communication management apparatus 50 searches the arrangementinformation management table (the arrangement information management DB5004, see FIG. 17B) using the site ID received in step S205 as a searchkey to read the arrangement information associated with the same site IDas the received site ID (step S206). Then, the transmitting/receivingunit 51 transmits the arrangement information read in step S206 to thecommunication terminal 70C (step S207). Thus, the transmitting/receivingunit (acquisition unit) 71 of the communication terminal 70C receivesthe arrangement information transmitted from the communicationmanagement apparatus 50.

Then, the selection unit 81 executes a process for selecting an imagecapturing device 10, based on the coordinate values indicated by thearrangement information read in step S207 and the number of divisionsindicated by the number-of-division information (step S208). Since theprocess for selecting an image capturing device 10 executed in step S208is similar to the processing of step S76 illustrated in FIG. 32 (thedetailed process is illustrated in the flowchart in FIG. 33), thedescription thereof will be omitted. The processing of steps S101 andS103 may be omitted. Then, in step S107, the storing and reading unit 79stores the predetermined-area information in the predetermined-areamanagement table (the predetermined-area management DB 7002, see FIG.20B) instead of the predetermined-area management table (thepredetermined-area management DB 5003, see FIG. 17A). Since theprocessing of steps S209 to S214 is similar to the processing of stepsS79 to S84 in FIG. 32, the description thereof will be omitted.

As described above, in the image communication system 1 according to amodification of the embodiment, the communication terminal 70 performs aprocess for selecting an image capturing device 10 that captures animage in which the point of interest P designated by a user isdisplayed. As a result, an image indicating the point of interest P canbe displayed on the communication terminal 70. Therefore, as in theembodiment described above, the image communication system 1 allows auser at a viewing site to view an image of the point of interest Pdesignated by the user.

As described above, the communication management apparatus 50 accordingto an embodiment of the present disclosure receives coordinate valuesindicating the position of the point of interest P in an area where aplurality of image capturing devices, the point of interest P beingdesignated by a user, and the number-of-division information fordividing the area from the point of interest P (step S75). Then, thecommunication management apparatus 50 transmits first predetermined-areainformation and second predetermined-area information to thecommunication terminal 70, the first predetermined-area informationindicating a predetermined area including a predetermined positioncorresponding to predetermined-position information in an image capturedby a reference image capturing device arranged at the shortest distancefrom the predetermined position among a plurality of image capturingdevices, the second predetermined-area information indicating apredetermined area including the predetermined position in an imagecaptured by a supplementary image capturing device arranged such thatthe angle between the supplementary image capturing device and thereference image capturing device is closest to an angle obtained bydividing the area in accordance with the number of divisions designatedby the user (step S77). Accordingly, the communication managementapparatus 50 can display an image of the point of interest P in adistribution site for which an image has been captured by an imagecapturing device selected in accordance with the number of divisionsdesignated by the user without performing a process on the imagecapturing device 10. Therefore, the communication management apparatus50 can display a captured image of an area including a predeterminedposition on the communication terminal 70 without causing a user toindividually operate a plurality of image capturing devices.

In addition, the communication management apparatus 50 can use a datatable that stores a coefficient for adding a weight that is set inadvance by the administrator in accordance with the number of divisionsdesignated by the user to remove an image capturing device that is notsuitable for use in image viewing among a plurality of image capturingdevices.

In the embodiment described above, as an example of a spherical image(spherical panoramic image), the captured image (entire image) is athree-dimensional panoramic image. However, the captured image (entireimage) may be a two-dimensional panoramic image.

In the embodiment described above, furthermore, the communicationterminal 70 at a viewing site may not be dedicated to viewing. Thecommunication terminal 70 may be configured to distribute a capturedimage and simultaneously implement both the distribution function andthe viewing function. Likewise, the distribution terminal 30 at adistribution site may not be dedicated to distribution. The distributionterminal 30 may be configured to display a captured image distributedfrom any other site and simultaneously implement both the distributionfunction and the viewing function. As described above, the imagecommunication system 1 may be configured to perform two-waycommunication of a captured image between a plurality of sites.

Each of the functions in the embodiment described above may beimplemented by one or more processing circuits or circuitry. In theembodiment described above, the term “processing circuit or circuitry”includes a device programmed to implement each function using software,such as a processor implemented by an electronic circuit. Examples ofthe device include an application specific integrated circuit (ASIC), adigital signal processor (DSP), a field programmable gate array (FPGA),a system on a chip (SOC), a graphics processing unit (GPU), and aconventional circuit module.

Further, various tables in the embodiment described above may begenerated by machine learning. Further, data of associated items can beclassified, such that use of tables can be optional. As used herein,machine learning is a technique that enables a computer to acquirehuman-like learning ability. Machine learning refers to a technology inwhich a computer autonomously generates an algorithm to be used fordetermination such as data identification from learning data loaded inadvance and applies the generated algorithm to new data to make aprediction. Any suitable learning method is applied for machinelearning, for example, any one of supervised learning, unsupervisedlearning, semi-supervised learning, reinforcement learning, and deeplearning, or a combination of two or more of those learning methods.

The above-described embodiments are illustrative and do not limit thepresent invention. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present invention.

Any one of the above-described operations may be performed in variousother ways, for example, in an order different from the one describedabove.

1. A communication management apparatus for communicating with acommunication terminal configured to display images captured by aplurality of image capturing devices, the communication managementapparatus comprising circuitry configured to: receive, from thecommunication terminal, predetermined-position information indicating apredetermined position in an area where the plurality of image capturingdevices are provided and number-of-division information indicating thenumber of divisions for dividing the area from the predeterminedposition; and transmit, to the communication terminal, firstpredetermined-area information indicating a predetermined area includingthe predetermined position in an image captured by a first imagecapturing device among the plurality of image capturing devices, andsecond predetermined-area information indicating a predetermined areaincluding the predetermined position in an image captured by a secondimage capturing device among the plurality of image capturing devices,the first image capturing device being arranged at a shortest distancefrom the predetermined position corresponding to thepredetermined-position information, and the second image capturingdevice being arranged such that an angle between the second imagecapturing device and the first image capturing device is closest to anangle obtained by dividing the area in accordance with the number ofdivisions indicated by the number-of-division information.
 2. Thecommunication management apparatus according to claim 1, wherein thecircuitry is further configured to generate predetermined-areainformation indicating predetermined areas including the predeterminedposition in images captured by in-range image capturing devices arrangedin a predetermined range from the predetermined position among theplurality of image capturing devices.
 3. The communication managementapparatus according to claim 2, wherein the circuitry is furtherconfigured to select the first image capturing device and the secondimage capturing device from among the in-range image capturing devices,based on the received predetermined-position information, arrangementposition information indicating arrangement positions of the pluralityof image capturing devices, and the received number-of-divisioninformation.
 4. The communication management apparatus according toclaim 3, wherein the number of divisions is denoted by n, with n being anatural number greater than or equal to 2, and wherein the circuitryselects, as a candidate of the second image capturing device, an imagecapturing device arranged at an angle closest to an angle obtained bydividing 360 degrees by the number of divisions n from a virtual linesegment connecting the first image capturing device and thepredetermined position.
 5. The communication management apparatusaccording to claim 3, wherein in response to a determination that anobstacle is present between the predetermined position and at least onein-range image capturing device among the in-range image capturingdevices, or in response to a determination that at least one in-rangeimage capturing device among the in-range image capturing devices has afailure, the circuitry removes the at least one in-range image capturingdevice from candidates of the second image capturing device.
 6. Thecommunication management apparatus according to claim 1, wherein thecaptured image is a spherical image.
 7. An image communication systemcomprising: the communication management apparatus according to claim 1;and a communication terminal configured to display images captured by aplurality of image capturing devices.
 8. The image communication systemaccording to claim 7, wherein the communication terminal includesanother circuitry configured to: control a display to display a flatimage indicating the area; receive input of the predetermined positionin the displayed flat image and input of the number of divisions; andtransmit the predetermined-position information and thenumber-of-division information to the communication managementapparatus, the predetermined-position information being informationindicating the predetermined position for which the input is accepted,the number-of-division information being information indicating thenumber of divisions for which the input is accepted.
 9. A communicationmanagement method executed by a communication management apparatus, thecommunication management apparatus being configured to communicate witha communication terminal configured to display images captured by aplurality of image capturing devices, the communication managementmethod comprising: receiving, from the communication terminal,predetermined-position information indicating a predetermined positionin an area where the plurality of image capturing devices are providedand number-of-division information indicating the number of divisionsfor dividing the area from the predetermined position; and transmitting,to the communication terminal, first predetermined-area informationindicating a predetermined area including the predetermined position inan image captured by a first image capturing device among the pluralityof image capturing devices, and second predetermined-area informationindicating a predetermined area including the predetermined position inan image captured by a second image capturing device among the pluralityof image capturing devices, the first image capturing device beingarranged at a shortest distance from the predetermined positioncorresponding to the predetermined-position information, and the secondimage capturing device being arranged such that an angle between thesecond image capturing device and the first image capturing device isclosest to an angle obtained by dividing the area in accordance with thenumber of divisions indicated by the number-of-division information. 10.Anon-transitory recording medium which, when executed by one or moreprocessors in a communication management apparatus, cause the processorsto perform a communication management method, the communicationmanagement apparatus being configured to communicate with acommunication terminal configured to display images captured by aplurality of image capturing devices, the communication managementmethod comprising: receiving, from the communication terminal,predetermined-position information indicating a predetermined positionin an area where the plurality of image capturing devices are providedand number-of-division information indicating the number of divisionsfor dividing the area from the predetermined position; and transmitting,to the communication terminal, first predetermined-area informationindicating a predetermined area including the predetermined position inan image captured by a first image capturing device among the pluralityof image capturing devices, and second predetermined-area informationindicating a predetermined area including the predetermined position inan image captured by a second image capturing device among the pluralityof image capturing devices, the first image capturing device beingarranged at a shortest distance from the predetermined positioncorresponding to the predetermined-position information, and the secondimage capturing device being arranged such that an angle between thesecond image capturing device and the first image capturing device isclosest to an angle obtained by dividing the area in accordance with thenumber of divisions indicated by the number-of-division information.